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FOREWORD 


The  effective  operational  test  and  evaluation  (OT&E)  of  defense  systems  is  a  critical  part  of  the 
long-term  program  to  jOTvide  for  the  proper  defense  for  the  Unit^  States.  The  Department  (rf 
Defense  h^  an  established  process  for  planning  and  conducting  operational  tests  and  for 
evaluating  the  data  that  result  from  those  tests. 

Volume  I  of  this  Operational  Suitability  Guide  was  prepared  by  the  Office  of  the  Director  of  Op¬ 
erational  Test  and  Evaluation  (DOT&E)  to  provide  an  overview  of  uiose  issues  that  ate  mclud^ 
in  the  general  subject  of  operational  suitability,  and  to  provide  background  information  for 
DCr&£  Staff  Assistants  to  use  when  exartuning  operational  suitability  subjects. 

Voliune  n  focuses  on  the  specific  activities  that  are  performed  by  the  DOT&E  Staff  Assistants. 
The  activities  that  ate  ad^ssed  are  the  review  of  the  Test  and  Evaluation  Master  Plans 
(TEMPs),  the  OT&E  plans,  and  the  OT&E  reports.  The  Volume  also  addresses  the  on-she 
observation  of  the  operational  testing  amivities.  The  information  in  the  Volume  n  is  intended  to 
supplement  the  policy  and  procedures  contamed  in  the  DoD  Directives  and  the  guidelines  that 
ate  contained  in  the  DoD  Manuals.  This  docmnent  does  net  establish  new  requirements  for 
operational  test  and  evaluation. 

Volume  n  is  orgattized  in  the  following  manner.  Chapter  1,  Introduction,  discusses  the 
arrangement  and  method  for  using  the  Volume.  Chapter  2  covers  the  requirements  for  Test  and 
Evaluation  Master  Plans  (TEMPs).  Chapter  3  addresses  the  content  of  OT&E  Plans.  Chapter  4 
contains  the  information  needed  for  observing  the  conduct  of  OT&E.  Chapter  5  contains  the 
information  needed  for  reviewing  OT&E  Reports. 

If  questions  or  commena  arise  while  reviewing  or  using  this  guide,  they  should  be  forwarded  to 
the  primary  author 


Dr.  Elizabeth  Rodriguez 
The  Office  of  the  Director  of  Operational  Test 
and  Evaluation 

Office  of  the  Secretary  of  Defense 
The  Pentagon,  Room  1C730 
Washington,  DC  20301 
(202)697-3895 


SIATHIDir  "A"  per  Dr.  E.  P.odriguez 
OSD/DOT&E,  The  Pentagon,  iJashington,  DC 
20301-1700 
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Chapter  1 
INTRODUCTION 


1.1  PURPOSE 

The  purpose  of  this  volume  of  the  Operauonal  Suitability  Guide  is  lo  assist  Staff  Assistants  in  the 
Office  of  the  Director  of  Operational  Test  and  Evaluation  (DOT&E)  in  their  review  of  OT&E 
documents  and  in  observing  OT&E  events.  It  highlights  important  operational  suitability  factors 
for  consideration  during  these  activities.  The  guide  is  structured  to  aid  in  the  review  and 
evaluation  of  the  three  principal  operational  test  and  evaluation  documents:  Test  and  Evaluation 
Master  Plans  (TEMPs),  OT&E  plans,  and  OT&E  reports. 

1.2  DEFINITIONS 

Operational  Test  and  Evaluation  is  conducted  to  determine  the  operational  effectiveness  and  suit¬ 
ability  of  weapons,  equipment,  or  munitions  for  use  in  combat  by  typical  milit^  users.  The 
Department  of  Defense  deflnes  operational  suitability  and  operational  effectiveness  in  the 
following  mannen 

Operational  Suitability:  “The  degree  to  which  a  system  can  be  placed  satisfactorily  in 

field  use  with  consideration  given  to  availability,  compatibility, 
transportability,  interoperability,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors,  manpower 
supponability,  logistics  supportability,  documentation,  and 
training  requirements.” 

Operational  EfTectiveness;  “The  overall  degree  of  mission  accomplishment  of  a  system 
when  used  by  representative  personnel  in  the  environment 
planned  or  expect^  for  operanonal  employment  of  the  system 
considering  organization,  doctrine,  tactics,  survivability,  vul¬ 
nerability,  and  threat  (including  countermeasures,  nuclear,  and 
chemical  and/or  biological  threais).” 

Operational  effectiveness  and  operational  suitability,  in  a  strict  sense,  cannot  be  separated.  There 
are  elements  within  each  that  easily  could  be  includ^  in  the  other. 

13  OVERVIEW 

This  guide  is  souctured  around  the  four  key  activities  of  the  staff  assistant:  the  three  documents 
DOT&E  must  review  (TEMPs,  plans,  and  reports)  and  the  observation  of  actual  testing,  with  the 
intent  of  focusing  on  specific  suitability  issues  that  will  be  reported  in  the  review  of  each.  The 
guide  is  organized  as  follows: 

Qiapter  2  contains  guidance  information  for  the  review  of  TEMPs. 

Chapter  3  contains  information  for  conducting  the  review  of  OT&E  Plans. 

Chapter  4  contains  information  for  observing  the  conduct  of  Operational  Testing. 

Chapter  S  contams  mfotmation  for  conducting  the  review  of  the  OT&E  Reports. 
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The  subsections  of  Chapters  2  through  S  consist  of  templates  that  ate  subdivided  into  the 
following  sections: 


The  Introduction  presents  an  overview  of  the  subject  of  the  template,  e.g.,  what 
the  particular  part  of  the  OT&E  document  should  contain. 

The  .^rea  of  Risk  identifies  the  risk  that  might  be  encountered  if  proper  test  plan¬ 
ning,  conduct,  or  reporting  of  suitability  concerns  is  not  followed. 

The  Outline  for  Reducing  Risk  directs  the  Sta^  Assistant  to  what  might  be 
included  in  a  proper  response  to  the  subject  of  the  template.  Each  of  the  items 
under  the  Outline  for  Reccing  Risk  is  further  emphasir^  by  having  an  example 
(enclosed  In  a  box)  of  its  application  to  a  DoD  system. 


1.4  HOW  TO  USE  THIS  GUIDE 

The  understanding  of  operational  suitability  and  the  meaning  of  the  suitability  elements  is  a  criti¬ 
cal  component  of  suitability  OT&E.  The  relationship  of  these  elements  to  each  other  and  to  the 
success^  introduction  of  the  system  into  the  operating  forces  are  important  and  necessary  parts 
of  system  acquisition. 

To  gain  full  value  from  the  information  within  this  guide,  it  is  recommended  that  the  reader  re¬ 
view  and  evaluate  the  complexity  of  operational  suitability  by  briefly  scanning  the  templates  in 
Chapters  2,  3,  4,  and  5.  A  thorough  mnt-to-back  study  of  the  document  will  then  provide  the 
reviewer  with  an  understanding  of  the  information  contained  in  the  guide,  thus  ^viding  a 
mental  road  map  for  quick  access  to  required  reference  material  when  and  as  it  is  needed. 

The  reader  may  wish  to  review  what  the  thirteen  elements  of  suitability  are,  what  parameters  are 
appropriate  to  measure  those  characteristics,  and  how  thresholds  for  diose  paranneters  might  be 
checked.  The  tutorial  volume  (Volume  I)  of  the  Operational  Suitability  Guide  can  be  reviewed 
for  diis  puipose. 

If  a  specific  document  is  under  review,  turn  to  the  appropriate  chapter:  Chapter  2  for  TEMPs, 
Chapter  3  for  OT&E  Plans,  or  Chapter  5  for  Test  Reports. 

If  information  relative  to  observing  Operational  Testing  is  required,  then  review  Chapter  4. 
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Chapter  2 

TEST  AND  EVALUATION  MASTER  PLAN  (TEMP) 


The  TEMP  is  an  essential  test  and  evaluation  (T&E)  document  used  by  the  Office  of  the 
Secretiuy  of  Defense  (OSD)  to  support  milestone  decisions  by  the  Defense  Acquisition  Board 
(DAB).  The  TEMP  is  the  basic  planning  document  for  all  T^  activity  related  to  a  pardcular 
system  acquisition.  It  defines  both  Developmental  Test  and  Evaluation  (DT&E)  and  O^radonal 
Test  and  Evaluation  (OT&E)  associated  with  system  development  and  acquisition  decisions. 
The  TEMP  relates  prognun  stmcture,  decision  milestones,  test  management  structure,  and 
required  resources  to  critical  operation^  issues,  critical  technical  issues,  evaluation  criteria  and 
procedures. 

One  of  the  mote  significant  functions  of  the  TEMP  is  to  document  test  and  evaluation  issues  and 
criteria  that  will  be  considered  in  acquisidon  decisions.  Thus,  the  reviewer  of  a  TEMP  must  real¬ 
ize  that  the  TEMP  serves  not  only  as  a  major  control  mechanism,  but  also  to  provide  a  clear 
correladon  between  issues  and  program  objecdves  through  test-verifiable  criteria.  In  reviewing  a 
TEMP  in  the  area  of  suitability,  one  must  ensure  that  it  contains  pertinent  suitability-related 
infoimadon  on  the  system's  required  operadonal  and  technical  cha^terisdcs,  test  objecdves, 
and  the  evaluadon  process. 

Suitability-related  system  requirements,  program  stmcture.  technical  ani  operadonal 
characteristics,  and  associated  thresholds  in  the  'TEMP  must  be  reviewed  to  ensure  that  they  are 
consistent  with  the  Requirements  Documents,  Acquisition  Decision  Memorandum  (ADM),  and 
approved  System  Concept  Paper  (SCP)/Decision  Coordinating  Paper  (DCP).  T^  must  be 
(fefined  sufficienUy  to  ensure  that  the  test  program  will  assess  the  effects  of  human  perfoitnance 
on  the  weapon  ^stem's  ability  to  meet  all  suitability  standards,  including  reliability  and 
maintainability. 

The  TEMP  must  include  the  system's  suitability-related  cridcal  techidcal  and  operadonal  issues 
and  diresholds  and  their  reladonship  to  the  system's  requirements.  It  should  clearly  oudine  the 
planned  T&E  process. 

A  TEMP  must  describe  the  kind  and  amount  of  suitability  test  and  evaluation,  required 
resources,  planned  test  locations  and  schedules.  It  must  clearly  relate  T&E  activity  to  suitability- 
related  critical  technical  characteristics  and  operational  suitability  issues.  It  must  describe  the 
evaluation  of  the  system  relative  to  the  suitability-related  issues  and  the  testing  to  be  conducted 
to  provide  data  to  accomplish  the  evaluation.  It  must  show  the  relationship  between  T&E 
schedules  and  program  decision  points  and  address  the  T&E  to  be  accomplished  in  each  program 
phase.  It  should  identify  the  planned  test  articles  to  satisfy  test  objectives,  as  well  as  identify  the 
number  and  rate  of  sysxms  to  be  produced  during  the  Low  Rate  Imtial  Production  (LRIP)  phase. 
Test  resource  requirements  must  be  addressed,  including  known  test  resource  shortfalls  that  may 
impede  the  full  test  and  evaluation  of  the  system  suitability. 

Finally,  in  reviewing  a  TEMP,  one  must  be  careful  to  avoid  the  pitfall  of  allowing  the  document 
to  become  an  end  unto  itself.  The  document  should  define  a  test  program  that,  when  ptopetiy 
executed,  will  provide  for  accurate  and  efficient  determination  of  a  weapon  system's  opetationtti 
effectiveness  and  suitability. 
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tA  PABTI,  SYSfiifl'DETftltS 

Part  Iota  TEMP  provides  details  of  the  weapon  system,  its  Intended  mission,  and 
the  required  technical  and  operational  characteristics.  The  mission  must  be 
adequately  defined  and  key  hardware  and  software  features  of  the  system  must 
be  described.  Technical  and  operaUonaf  characteristics  discussions  must  include 
their  relationship  to  the  suitability  aspects  of  system  performance,  as  well  as  the 
effectiveness  aspects. 

AREA  OF  RISK 

Heavy  emphasis  on  effectiveness  requirements  may  lead  to  oversights  in  suitability  require¬ 
ments. 

The  suitability  implications  of  speafied  mission  definitions  and  the  system’s  technical  and 
operational  charactensocs  often  are  not  well  understood  or  clearly  visible.  During  the  early 
stages  of  a  program,  there  is  a  tendency  to  place  heavy  emphasis  on  the  detinition  and  under- 
statiding  of  operational  effectiveness.  Later  in  the  program,  suitability  deficiencies  that  signifi¬ 
cantly  detract  from  system  capability  are  identified. 

OUTLINE  FOR  REDUCING  RISK 

The  system's  operational,  support,  and  maintenance  concepts  should  be  examined  to  identify 
impoitant  suitability  considmtions,  as  well  as  the  testability  of  key  characteristics  and 
requirements. 

a.  Mission  Description.  (Sec  2.1.1) 

Can  key  operational  suitability  issues  be  identified  from  the  mission  description  in  the 
TEMP? 

The  mission  of  the  system  and  the  planned  suppon  concept  should  be  described  in  enough  detail 
to  peimit  the  reviewers  and  the  decisionmakers  to  understand  the  critical  operation^  issues 
(COIs),  including  the  suitability  COls.  If  the  mission  contains  new  or  unique  requirements,  these 
^ould  be  e.xamined  to  ensure  that  the  test  program  has  the  structure  to  deal  with  these  features. 


The  mission  of  a  weapon  system  is  to  penetrate  enemy  defenses  and  to  conduct 
offensive  actions  behind  enemy  lines  in  an  independent  and  self-sustaining  mode 
for  a  period  of  60  days.  Cnfical  operational  issues  must  address  the  suitability 
areas  of  reliability,  maintainability,,  and  graceful  degradation  of  the  system  as 
required  by  the  self-sustaining  nature  of  the  system  s  mission 
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b.  System  Description.  (Sec.  2.1.2) 

Are  the  key  system  features  that  drive  the  suitability  requirements  included? 

Critical  operational  suitability-related  system  characteristics  may  be  omitted,  resulting  in  inade¬ 
quate  planning  for  that  pomon  of  OT&E.  The  suitability  issues  may  be  improperly  identified;  as 
a  result,  a  critical  issue  will  be  missed.  Inadequate  attention  might  be  placed  on  the  operational 
suitability  issues. 


A  key  feature  of  the  missile  warning  system  (MWS)  is  improved  operational 
availability.  With  the  use  of  fault-tolerant  computer  hardware  and  software,  the 
system  R&M  will  significantly  improve  end-to-end  availability  for  the  MWS. 
The  improved  R&M  is  cntical  to  meet  the  high  level  of  operational  availability. 


c.  Critical  Technical  Characteristics.  (Sec.  2.13) 

Do  the  technical  characteristics  support  the  operational  suitability  requirements? 

The  key  technical  characteristics  should  be  described  clearly.  The  rationale  for  each  identified 
critical  technical  characteristic  is  important  in  understanding  how  the  technical  testing  program 
fits  into  the  overall  acquisition  program.  This  knowledge  sirould  then  be  used  to  understand  the 
relationship  of  the  technical  testing  to  the  key  operation^  suitability  requirements. 


The  system  has  tour  critical  areas  of  technical  performance.  The  classified 
appendix  lists  these  characteristics,  and  quantitative  measures  against  test 
iocations,  schedule  and  the  decisions  supported,  which  provide  the  technical  basis 
against  which  the  system  performance  wifi  be  evaluated.  These  four  areas  are 
undetected  message  error  rate,  operational  availability,  processing  time  and 
system  growth. 


d.  Required  Operational  Characteristics.  (See  2.1.41 

Are  the  required  operational  characteristics  and  their  associated  parameters  listed? 

Tlir.  operational  characteristics-with  associated  operational  effectiveness  and  suitability 
parameteis~tha>  are  critical  to  the  mission  performance  and  the  ability  to  place  the  system  into 
field  use  should  be  listed  Thresholds,  which  represent  the  level  of  ^stem  performance  accept¬ 
able  to  the  user  to  successfully  execute  the  mission,  also  should  be  listed.  The  Service-prepared 
TEMP  identifies  the  key  required  operational  characteristics  needed  for  operational  mission 
accotnplishmenL  One  must  ensure  consideration  is  given  to  the  testability  of  the  requirements, 
utilizing  the  expertise  of  the  development  and  test  and  evaluation  communities,  and  check  that 
the  key  required  operational  suitability  characteristics  deemed  of  critical  importance  to  meeting 
the  mission  requirements  are  identified. 


The  key  required  operational  characteristics  of  the  system  include  the  following 
suitability  requirements,  system  reliability,  system  maintainability,  and 
logistics  supportability.  The  values  are  contained  in  the  classified  appendix. 
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The  Mission  Description  section  of  the  TEMP  should  brielly  describe  the  mission 
of  the  deployed  system,  the  threat  it  is  required  to  be  effective  against,  any 
threats  that  must  be  countered  during  the  accomplishment  of  the  mission,  and 
the  range  of  environmental  conditions  (weather,  terrain,  oceanographic,  space) 
over  which  the  system  should  be  effective  and  suitable.  As  necessary.  It  should 
reference  other  appropriate,  approved  program  documents.  This  ndght  Include 
the  Mission  Need  Statement  (MNS),  System  Concept  Paper  (SCP),  Decision 
Coordinating  Paper  (DCP),  or  Service  need  statements,  such  as  the  Required 
Operational  Caf^Hty  Statement  (ROC)  or  Statement  of  Operational  Need  (SON). 

AREA  OF  RISK 

The  test  program  may  not  be  structured  to  evaluate  the  system  for  its  actual  operational 
mission. 

If  the  mission  profiles  are  not  clearly  defined  and  understood  by  both  the  user  and  the  develcqter 
early  in  the  acquisition  cycle,  not  only  may  the  analyses  and  design  activities  be  jetqtardized,  but 
also  the  test  program  may  be  plann^  in  a  manner  that  is  not  fully  relevant  to  its  intended  use. 
In  the  suit^ility  area,  this  mission  definition  needs  to  identify  any  specific  reliability, 
maintaiMbility,  logistics,  or  other  suitability  area  tfiat  has  a  critical  relationship  to  the  successful 
accoii^lishment  of  the  system's  mission. 

OUTLINE  FOR  REDUCING  RISK 

Reducing  the  risk  associated  widi  the  mission  description  and  the  associated  suitability  issues 
quires  that  the  system's  operational  and  suppon  concepts  be  examined.  Those  facu»s  that  are 
iitqxrttant  suitability  issues  should  be  identified  and  doemnented  in  the  test  and  evaluation  master 
plan. 

a.  Missioa  Definition. 

Does  the  mission  description  include  items  that  could  be  suitability  issues? 

The  mission  ot  tmssions  that  are  planned  for  the  system  should  be  described  in  sufficient  detail 
to  ensure  that  critical  operational  suitability  issues  can  be  identified,  and  the  context  of  these 
issues  understood.  The  TEMP  should  be  compared  to  other  program  documentation  to  ensure 
that  it  agrees  with  the  previous  mission  descriptions.  If  the  detail  is  inadequate,  then  the  need  for 
addition^  detail  should  be  judged,  based  upon  what  information  is  avail^Ie  from  other  sources 
and  if  this  information  is  in  documentatioa  that  is  controlled  as  to  content 

The  GBU-15  is  a  modular,  unpowered,  air-to-suiface  guided  munition  designed 
for  external  carriage  on  F-4  and  F-111  aircraft.  It  is  designed  for  employment 
from  low  and  high  altitude  and  appropriate  standoff  ranges  against  high  vaiue 
targets.  The  addition  of  an  infrared  guidance  section  provides  day/night  capability 
as  well  as  limited  adverse  weather  capabilities.  Suitability  areas  of  particular 
importance  include;  maintenance  concepts,  envuonmental  considerations,  safety, 
and  transportation  (including  handling). 


b.  Environmeiifad  Conditkms. 


Is  the  range  of  environmental  conditioas,  over  which  the  system  will  be  effective, 
identified? 

The  mission  description  should  include  a  discussion  of  the  range  of  environmental  conditions 
over  which  the  system  must  be  effective.  This  should  include  weather,  oceanographic,  space, 
tenain,  obscurants,  vegetation,  illumination,  etc.,  as  is  appropriate  for  ^  pardcular  system  in 
question. 


The  vehicle  loaded  to  the  *<ull  combat  load*  gross  weight  should  be  able  to  ford 
water  to  the  depth  of  four  feet  and  cnmb  a  30  degree  incline.  The  low-light-level 
rangefinder  should  be  able  to  support  the  scout  mission  under  the  ambient 
nighttime  conditions  that  are  representative  of  conditions  in  Europe  and  the  Mid¬ 
dle  East. 


c.  Definition  of  Logistic  Support  Concept 

Is  the  logistic  support  concept  defined  in  sufficient  detail  to  allow  the  planning  of  the 
operation  suitability  portion  of  the  OT&E? 

The  plamied  logistic  support  concept  for  die  system  may  Rqniic  that  unique  aspects  or 
approaches  for  ^  support  of  the  system  be  realfeed.  The  mission  desetiptioo  in  the  TEMP 
should  highlight  any  of  these  aspects  that  arc  unusual  for  die  type  of  system  being  discussed. 
The  need  here  is  to  identify  any  unique  aspects  of  the  support  scenario  or  cow^t  in  enough 
detail  so  that  die  test  Ingram  can  be  plann^  to  addicss  dir^  aspects.  The  detail  must  be  at  a 
level  that  shows  the  unique  aspects  and  gives  die  planners  a  foundation  to  structuie  the  OT&E. 


The  F-XX  is  a  light  attack  aircraft  vnth  leap-ahead  combat  effectiveness  and 
battlefield  survi''ability  to  defeat  the  threat  of  the  mid-1990s.  The  F-XX  has  a 
woridtinde  operational  capability  and  high  sortie  generation  rates  greater  than  5 
sorties  per  day  in  sustained  operation.  It  will  perform  combat  tasks  in  the  close- 
in.  deep,  and  rear  battle  environments.  The  support  concept  for  the  aircraft  is  a 
three-level  concept.  In  contrast,  major  avionics  items  have  the  reliability  and 
maintainability  required  to  be  supported  using  two  levels  of  maintenance.  (Avi¬ 
onics  items  to  be  supported  by  two  levels  of  maintenance  are  listed  in  table  V-1.) 
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The  System  Description  section  of  the  TEUP  briefly  describe  the  system's 
design  Including  key  features  and  unique  characteristics,  interfaces  with  other 
systems,  and  unique  support  concepts.  Key  features  and  subsystems  should 
Include  both  hardware  and  software  elements,  as  appropriate.  Unique 
rdraracteristlcs  or  unique  support  concepts  should  be  identilied  If  they  result  in 
the  requirement  for  special  test  or  anises  during  test  and  evaluation.  The 
relative  maturity,  integration,  and  modification  requirements  of  any  Non- 
Developmental  Items  (NO!)  should  be  addressed.  Interfaces  with  existing  or 
planned  systems  that  are  required  for  mission  accomplishment  should  be 
identified.  Any  interoperability  wiUt  existing  and/or  planned  systettK  of  other 
DoD  Components  or  allies  should  be  identitied. 

AREA  OF  RISK 

A  pwMT  system  description  may  lead  to  improperiy  identified  suitability  issues  or  missed 
critical  issues. 

The  significance  of  the  System  Description,  relative  to  operational  suitability,  is  that  critical 
system  characteristics  may  be  omitted  and  the  requirements  for  the  operational  suitability  portion 
of  OT&E  may  be  impropCTly  platuied.  The  suitability  issues  may  be  improperly  identifi^  as  a 
result,  a  critical  issue  will  be  missed.  Test  planners  and  the  decisionmakers  might  place 
inadequate  attention  on  the  operational  suitability  issues. 

OUTLINE  FOR  REDUCING  RISK 

It  is  necessary  to  review  the  supporting  documentation  aixl  ensure  that  the  system  description 
adequately  describes  the  system’s  operational  suitability  features,  and  what  is  different  about  dris 
system. 

a.  Key  Features  Description. 

Docs  the  description  of  key  features  of  the  system  and  its  subsystems  include  those  that  re¬ 
late  to  the  operational  suitability  issues? 

Key  features  and  subsystems  ate  those  that  allow  the  system  to  perform  its  requited  t^rerational 
mission.  The  descriptions  of  the  key  fearores  and  subsystems  provide  a  basis  of  information  for 
assessing  the  adequacy  of  die  OT&E  program  that  is  dekribed  in  the  TEMP. 


The  mortar  consists  of  a  55mm  tube,  a  telescopic  sight,  and  a  two-piece 
baseplate.  Eight  are  issued  to  20-man  mortar  platoons.  The  mortar  is  designed 
to  be  hand-cam'ed  by  two  men.  The  ammunition  weighs  25  lbs.  per  round  The 
platoon  will  have  sufficient  HMMWV  vehicles  to  transport  every  mortar  plus  its 
basic  load  of  ammunition  (50  rounds). 
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b.  Relationship  with  Existing  or  Planned  Systeni(s). 

Are  the  interfaces  with  existing  systems  identified? 

The  identification  of  the  interfaces  with  other  existing  or  planned  systems  results  in  a  list  of 
potential  requirements  for  interoperability  and/or  compatibility  issues.  This  list  forms  the  basis 
for  systems  that  must  be  acquir^  for  use  during  the  operational  testing.  If  the  systems  are  not 
available  when  required,  then  the  potential  for  a  serious  test  limitation  must  be  examined.  The 
electromagnetic  envirotunental  effects  (E3)  area  can  be  a  significant  issue  for  some  systems. 
Having  complete  knowledge  of  all  other  systems  that  will  be  operating  in  proximity  to  the 
system  under  test  will  help  in  defining  the  test  objectives. 


Two  communications  systems,  each  with  a  different  mission,  entered  development 
at  approximately  the  same  time  The  first  was  subjected  to  operational  testing 
without  the  second  system  being  present  When  a  joint  test  finally  was  conducted, 
there  was  a  serious  incompatibility  problem  between  the  two  systems.  It  was  not 
possible  to  operate  the  first  system  unless  the  second  system  was  turned  off. 


c.  Unique  Characteristics  of  the  SysteriL 

Are  unique  characteristics  of  the  system  or  unique  support  concepts  identified? 

Characteristics  that  ate  unique,  different,  or  better  in  relation  to  operational  suitability  should  be 
identified.  Any  unique  suitability  characteristics  should  be  considered  when  the  suitability  COIs 
are  defined.  To  form  a  basis  for  this  consideration,  these  unique  characteristics  should  be  listed 
in  the  System  Description.  Also,  these  unique  characteristics  or  support  concepts  should  be 
consider^  when  planning  for  the  opeational  test  events.  Having  an  ad^uatc  dcsrapiitMi  in  the 
TEMP  allows  the  remaining  planning  to  proceed. 


A  missile  system  has  a  unique  Go/No-Go  test  to  indicate  the  status  of  the  system. 
It  is  important  that  the  test  be  designed  to  clearly  isolate  a  fault  to  the  missile  or 
the  launching  system  Failure  to  isolate  the  fault  in  this  manner  will  result  in 
delayed  or  ineffective  missions  or  the  need  for  additional  maintenance  capability. 
The  TEMP  describes  these  features  in  sufficient  detail  to  ensure  such  unique  tests 
are  addressed  in  the  test  plan 


d.  Changes  over  Previous  System. 

Are  there  significant  changes  or  improvements  over  the  predecessor  system? 

Major  improvements  in  performance  over  the  predecessor  system  can  result  in  risk  areas  that 
should  be  the  focus  of  operational  testing.  If  such  improvements  are  predicted  in  some  of  the 
suitability  elements,  the  remaining  elements  will  be  adjusted  to  remain  in  balance.  Thus,  the 
major  improvement  must  be  confirmed  during  OT  if  the  system  is  to  be  operationally  suitable. 


An  electronics  system  using  new  technology  electronic  components  is  forecasted 
to  have  a  Mean  Time  Between  Failure  that  is  seven  times  that  ot  the  system  it  is 
replacing  The  maintenance  manpower  and  the  logistics  support  that  is  planned 
are  significantly  less  than  that  of  the  predecessor  system.  The  reliability  of  the 
system  is  identified  as  a  COI. 
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The  Critical  Technical  Characteristics  section  identifies  the  technical 
characteristics  whose  measurements  are  the  principal  indicators  of  the  system’s 
technical  achievement.  The  section  should  identify  performance  thresholds  and 
the  milestones  at  which  each  of  the  thresholds  Aas  been,  or  is  scheduled  to  be, 
demonstrated.  Characteristics  should  be  quantified  when  possible.  Other  pro¬ 
gram  documerris  may  be  referenced  for  the  technical  characteristics,  parUculariy 
if  the  technical  requirements  are  classified.  Technical  characteristics  usually  are 
contract  specifications  and  are  derived  from  operational  user  requirements  which 
precipitated  the  need  lor  the  system.  Therefore,  they  may  be  traceable  to  the 
required  operational  effectiveness  and  suitability  characteristics. 

AREA  OF  RISK 

Poor  definition  of  critical  technical  characteristics  can  lead  to  a  test  program  that  is  unable 
to  assess  program  risk. 

TEMPs  frequently  do  not  demonstrate  the  relationship  between  the  technical  chaiacteristics  and 
the  operational  chaiacteristics,  including  operational  suitability.  The  technical  thresholds 
provided  oftentiines  are  presented  without  a  clear  justification  to  their  choice  m  an  explanation 
of  their  significance  or  testabili^  prior  to  the  intended  milestone.  The  purpose  of  techniod 
testing  is  to  reduce  risk  by  assuring  fiiat  a  portion  of  the  challenge  of  achieving  the  operational 
requirements  has  been  met  To  understand  what  degree  of  risk  has  been  reduceth  the  relationship 
between  the  technical  characteristics  and  the  operationai  requiiements  must  be  discussed. 

OUTLINE  FOR  REDUCING  RISK 

Having  an  understanding  of  the  technical  characteristics  and  the  rationale  or  specific  reference 
relating  each  identified  critical  technical  characteristic  is  important  to  understanding  how  the 
techni^  testing  program  fits  into  the  overall  acquisition  program.  This  knowledge  ^ould  then 
be  used  to  undoistand  the  relationship  to  the  requimd  operational  suitability  capability. 

a.  Critical  Characteristics  Listing. 

Are  the  key  technical  hardware  and  software  characteristics  and  thresholds  listed? 

Key  technical  characteristics  ate  found  in  the  system  specifications  and  requirement  documents. 
The  TEMP  should  summarize  them  and  relate  them  to  the  milestones  in  a  matrix  format  The 
matrix  should  indicate  the  characteristics  that  have  been  evaluated  or  that  will  be  evaluated 
during  the  remaining  phases  of  the  developmental  test  (DT).  Each  technical  characteristic  should 
have  a  threshold  vidue.  If  the  technical  characteristics  are  summarized  m  another  acquisition 
document,  such  as  a  Baseline  Coirelaticn  Matrix  (BCM),  this  document  should  be  compared  to 
the  TEMP.  The  technical  characteristics  that  relate  to  opeiationa]  suitability  generally  are  quan¬ 
tifiable  items,  such  as  reliability  and  maintainability.  One  should  examine  these  chaiacteristics 
and  determine  how  they  relate  to  the  opciational  suitability  characteristics.  What  is  their  basis'’ 
Are  they  engineering  estimates  of  expected  performance  required  by  contract  specifications? 
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Since  technical  chatacteristics  often  ate  measuied  in  a  mote  pristine  manner  than  exists  in  the 
operational  units,  the  reliability  and  maintainability  characteristics  tend  to  be  more  optimistic 
than  the  operational  characteristics.  They  should  be  examined  to  determine  if  the  proper 
relationship  exists. 
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b.  Milestone  Inte.-vals. 


Are  milestones  for  demonstration  of  key  suitability-related  tecbnical  thresholds  identified? 

The  technical  thresholds  should  be  tied  to  specific  milestones.  At  those  milestones,  sufficient 
testing  of  the  respective  technical  areas  should  have  been  accomplished  and  the  r^ts  submitted 
to  the  decisiommdcets.  This  schedule  also  should  be  compatiWe  with  the  operational  T&E  of  the 
companion  operational  suitability  characteristics.  The  program  milestone  documentation  should 
be  checked  to  ensure  that  the  technical  characteristics  that  ate  important  to  critical  suitability 
issues  ate  addressed  on  a  schedule  consistent  with  the  rest  of  the  acquisition  program. 


The  Technical  Characteristics  listing  indicates  the  decision  that  is  supported  by 
each  of  the  technical  thresholds.  Phases  of  testing  are  indicated  that  wiii  yieid 
information  to  judge  the  system  against  the  thresholds  at  each  decision  point. 


c.  Suitability  Characteristics. 

Do  the  technical  suitability  characteristics  describe  minimum  acceptable  system  perfor¬ 
mance? 

The  technical  chatacteristics  should  be  quantifiable,  where  appropriate,  related  to  system 
operational  suitability  characteristics,  and  provide  the  quantitative  measures  against  which 
system  performance  will  be  evaluated  during  system-level  technical  testing.  Technical  suitabili¬ 
ty  characteristics  usually  are  defined  differently  than  operational  characteristics,  and,  as  such,  re- 
^t  in  higher  values. 


The  Technical  Characteristics  listing  indicates  that  the  diagnostics  false  alarm 
rate  threshold  is  2  percent  of  all  system  test  failures.  For  two  BIT  actuations, 
the  talse  alarm  rate  shall  be  less  that  0.1  percent.  Diagnostics  between  different 
levels  of  testing  shall  incorporate  tolerance  levels  that  preclude  retest-OK  and 
cannot  duplicate  anomaBes  above  the  2  percent  rate. 
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This  section  of  the  TEMP  shoutd  tist  the  key  operationai  characteristics  and  their 
associated  parameters  as  identified  in  the  user's  need  statement  and  approved  by 
the  Service.  A  key  operationai  characteristic  is  a  principai  eiement  of  the 
system's  abiiity  to  accompiish  its  mission  (operationai  effectiveness)  and  to  be 
supported  (operationai  suitabiiity).  These  characteristics  usuaiiy  are  defined  by 
parameters  that  are  indicators  of  the  system’s  abiiity  to  accompiish  its  mission.  If 
the  Service  has  an  approved  mabix  of  operational  requirements  In  the 
requirements  documents,  It  may  be  used  to  display  the  required  operational 
characteristics. 

AREA  OF  RISK 

Key  operational  characteristics  may  not  be  highlighted  and  the  planned  testing  may  not  be 
sufficient  to  ensure  achievement  of  the  mission  requirement 

Key  required  operational  characteristics  needed  for  operational  mission  accomplishment  may  ad¬ 
dress  attention  to  the  deficiencies  of  the  current  system.  While  characteristics  often  are  drawn 
from  Service-approved  formal  documents,  the  danger  exists  that  the  formal  document  may  be 
incomplete  or  may  be  written  in  generalized,  non-quantitative  terms.  Consequently,  the 
operational  requuements  parameters  selected  for  highlighting  may  not  be  sufficient  to  ensure 
achievement  of  the  mission  requirement  Since  the  operational  suitability  characteristics  are  seen 
by  many  people  as  secondary  considerations,  there  needs  to  be  assurance  that  adequate  descrip¬ 
tion  and  definition  of  the  these  items  are  included  in  the  TEMP. 

OUTLINE  FOR  REDUCING  RISK 

Key  required  suitability  characteristics  that  are  of  critical  importance  to  meeting  the  mission 
requirements  should  be  identified.  The  description  of  each  key  suitability  characteristic  should 
include  the  appropriate  parameters  and  thresholds. 

a.  Operational  Suitability  Requirements. 

Are  all  operational  suitability  requirements  identified? 

Suitability  issues  are  identified  in  the  TEMP.  The  COls  should  cover  all  suitability  elements; 
the  reason  for  eliminating  any  element  should  be  discussed.  Other  operational  requirements 
might  be  in  the  diagnostics  area  or  in  levels  of  support  needs,  e.g.,  numbers  of  maintenance 
people,  test  equipment,  levels  of  maintenance,  software  change  requirements.  In  reviewing  the 
TEI^,  the  key  is  to  ensure  that  the  cnucal  suitability  areas  are  defined  by  operational 
requirements.  Definitions  should  mclude  all  operationally  relevant  situations;  e.g.,  all  failures 
that  can  occur  in  operational  use  should  be  included  in  the  definition  for  the  operational  suitabili¬ 
ty  characteristics  and  parameters.  This  information  should  be  a  direct  follow-on  tiom  the 
system's  user  requirements.  It  is  very  difficult  for  the  operational  test  community  to  plan  and 
conduct  a  meaningful  operational  test  if  the  user  community  has  not  completely  defined  realistic 
quantitative  needs. 
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The  reliability  of  the  communications  system  should  be  indicated  by  the  mean 
time  between  operational  mission  failure  (MTBOMF)  values  demonstrated  The 
user  equipment-requirement  MTBOMF  must  be  greater  than  500  hours  for  both 
the  manpack  and  the  aviation  sets  The  test  is  planned  to  demonstrate  an  80 
percent  confidence  in  the  MTBOMF  values. 


b.  Parameters  and  Thresholds. 

Are  suitabOity  characteristics  supported  by  parameters  and  thresholds? 

Each  of  the  key  opeiationai  characteristics  should  have  an  associated  parameter  and  the 
parameters  should  be  accompanied  by  a  threshold  (usually  but  not  always  quantitative).  The 
thresholds  should  not  be  engineering  estimates  of  expected  performance  or  the  performance  that 
is  required  by  contrartt  specifications;  they  should  be  the  minimum  system  performance 
acceptable  to  the  user  to  successfully  execute  the  mission.  If  only  a  component  of  a  system  is 
being  developed,  the  thresholds  should  represent  the  essential  operational  requirements  of  the 
total  weapon  system.  Failure  derinitions  should  be  included  by  reference,  and  should  reflect  all 
failure  m^s  or  events  that  can  be  expected  to  occur  in  operational  service. 


REQUIRBJ  OPERATIONAL  CHARACTERISTICS 
WITH  THEIR  ASSOCIATED  PARAMETERS  AND  THRESHOLDS 


CHARACTERISTIC 

PARAMETER 

THRESHOLD 

Reliability 

Mean  time  between  operational 
mission  failures  (MTBOMF) 

Mean  lime  between  maintenance 

500  hrs. 

actions  (MTBMA) 

32  hrs. 

Maintainability 

Maintenance  manhours  per 
operating  hour 

2.5  hrs. 

c  Qualitative  Suitability  Requirements. 

Are  critical  suitability  areas  expressed  qualitatively? 

With  some  systems,  there  are  critical  aspects  of  the  system’s  suitability  performance  that  can 
only  be  expressed  qualitatively.  There  should  be  no  reluctance  to  include  such  items  in  the  list  of 
required  o^rationiil  characteristics.  The  items  that  are  critical  to  the  system’s  ability  to  perform 
its  mission  within  the  intended  field  environment  may  require  the  attainment  of  objectives  that 
are  ducribable  only  in  a  qualitative  manner.  This  could  rnclude  requirements  for  compatibility 
with  (Xher  systems  or  with  certain  skill  level  operating  and  maintenance  personnel,  or  require¬ 
ments  that  are  in  areas  that  have  not  progressed  to  the  level  where  the  requirement  can  be  accu¬ 
rately  quantified. 


The  system  must  be  air-transportable  This  requires  that  the  system  must  be 
able  to  be  prepared  for  air  shipment  by  organizational-level  personnel  and  the 
system  be  air-transportable  by  C-130,  C-141,  and  C-5  aircraft. 
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Part  II  of  a  TEMP  identifies  responsibilities  of  the  participating  organizations,  as 
weli  as  the  management  and  scheduling  aspects  of  the  weapon  system  program. 
Key  factors  to  be  discussed  include  TAE  and  acquisition  strategy  relationships, 
key  decision  points  and  associated  reports,  TAE  requirements  to  support  UUP, 
constraints,  test  article  and  critical  support  resource  availability,  and  the 
associated  TAE  responsibilities  of  all  participating  organizations. 

AREA  OF  RISK 

Suitability  testing  will  not  be  perfoimed  at  a  time  to  support  the  program’s  dedsioii  mile¬ 
stones. 

There  is  significant  program  risk  if  the  suitability  of  the  system  is  not  demonstrated  in  time  to 
support  decision  milestones.  Typically,  the  operational  suitability  aspects  of  a  weapon  system 
program  lag  the  effectiveness  aspects  of  the  program.  Design  of  support  equipment  cannot 
commence  until  the  design  of  the  actual  weapon  system  begins  to  stabilize.  Maintenance 
manuals  catuiot  be  written  until  the  support  equipment  is  defined.  As  a  result  of  this  inherent 
lag,  actual  testing  of  the  suitability  aspects  trxrre  often  than  not  does  not  occur  in  concert  with 
testing  the  effectiveness  of  the  weapons  system.  If  not  appropriately  planned  for  and  scheduled, 
critic^  suitability  elements,  including  (hagnostics  capabilities,  support  equipment,  techtucal 
ttranuals,  etc.,  may  not  be  available.  Therefme,  suitability  testing  will  not  be  performed  in  a 
timely  manner  to  support  the  program’s  decision  milestones. 

OUTLINE  FOR  REDUCING  RISK 

Management  responsibilities  must  be  clearly  delineated  and  include  etrrphasis  on  the  suitabili^ 
aspects  of  the  planned  test  program.  The  integrated  schedule  should  indicate  that  the  test  pro¬ 
gram  is  not  schedule  driven,  but  accomplishment  driven. 

a.  Integrated  Schedule,  (see  2.2.1) 

Are  all  the  required  suitability  activities  dearly  identified,  appropriately  fime-phased,  and 
adequately  resourced  so  as  to  provide  the  required  results  at  the  decision  milestones? 

The  integrated  schedule  should  demonstrate  that  suitability  testing  will  be  consistent  with  the 
need  for  information  at  the  key  program  decision  milestones.  The  relationship  of  the  test  periods 
to  the  schedule  should  give  an  idea  of  the  amount  of  test  time  that  will  be  available  to  support  the 
various  milestone  decisions  and  the  associated  reporting  requirements.  The  integrated  schedule 
should  give  an  indication  of  when  the  various  suitability  elements  will  be  available  for  OT&E; 
this  includes  when  the  support  equipment  and  representative  maintenance  personnel  will  become 
available  for  the  various  levels  of  support,  etc.  The  placement  of  these  items  on  the  integrated 
schedule  will  give  the  reviewer  an  idM  of  when  various  portions  of  the  operational  suit^ility 
evaluation  will  be  done  and  when  the  results  of  this  activity  will  bKome  available  for 
presentation  to  the  decisionmakers.  One  must  review  the  schedule  to  ensure  the  needed  support 
to  obtain  suitability  objectives  has  been  idenufiet^  this  includes  testing  and  associated  hardware 
and  facility  requirements  down  to  the  lowest  practical  level. 
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The  integrated  schedule  for  the  M165  Truck  program  displays  'on-dock'  dates 
for  the  test  articles  to  be  used  for  maintrnance  tear-down  and  test  personnel 
training.  The  sequencing  of  the  events  indicates  that  the  required  support 
elements  will  be  available  in  a  timely  manner  to  support  operational  testing 


b.  Management  (see  2.2..2) 

Are  responsibilities  for  the  operational  suitability  areas  clearly  identified  and  appropriately 
assigned? 

One  must  ensure  that  the  responsibilities  are  depicted,  giving  suitability  objectives  the  needed 
attention,  and  that  a  sufficient  number  of  test  events  are  scheduled  to  provide  a  level  of 
confidence  in  the  resulting  suitability  statistics.  The  relationship  between  the  key  decision  points 
and  the  specific  T&E  reports  also  should  be  examined. 


Overall  responslullity  of  the  LX-21  Helicopter  program  Is  the  responsibility  of 
the  Program  Manager  (PM).  The  PM  establishes  and  chairs  the  TIWG  and  assures 
that  adequate  technical  testing  is  accomplished.  TRADOC  represents  the  user  and 
also  is  responsible  for  ensuring  that  the  t'aining  of  all  test  players  meets  or 
exceeds  the  minimum  acceptable  standards  used  to  measure  training  effective¬ 
ness.  Technical  testing  under  TECOM's  direction,  includes...Reliability, 
Availability  and  Maintainability  (RAM)  measures.  AVSCOM  .-tlso  is  responsible 
for  management  oversight...to  assure  ..RAM  maturity  and  growth  Is  continuing  as 
it  pertains  to  flight  handling  qualities  and  airworthiness.  U.S.  Army  OTEA  will 
conduct  Continuous  Otmprehensive  Evaluation  and  the  initial  OT&E  using  LRIP 
aircraft  USALEA  is  responsible  for.. .ensuring  appropriate  logistics  testing  and 
evaluation  are  accomplished. 
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The  entire  program  schedule  should  be  displayed  on  one  page  to  Include  the 
Integrated  time  sequencing  of  critical  T&E  phases  or  events,  related  activities, 
and  planned  cumulative  furrding  expenditures  by  appropriation.  Included  should 
be  event  dates  such  as  program  decision  milestones,  test  article  availability,  criti¬ 
cal  support  resource  availability;  appropriate  phases  of  DT&E,  live  fire  T&E  end 
OT&E;  rate  production  deliveries  (i.e.,  low  and  full);  In'ttial  Operatiorral  Capability 
(IOC);  Full  Operational  Capability  (FOC);  Low  Rate  Initial  Production  (LRIP);  and 
beyond  LRIP. 

AREA  OF  RISK 

bsuflident  time  and  resources  may  be  planned  for  suitability  testing. 

The  integrated  schedule  may  not  cover  all  the  necessary  events  or  dates  to  communicate  all  of 
the  asi^ts  of  system  testing.  The  TEMP  integrated  schedule  may  be  at  such  a  sumrnary  level 
that  it  is  difficult  to  determine  the  time  and  resotiices  available  for  the  suitability  portion  of  the 
OT&E.  Without  this  level  of  visibility,  it  might  not  be  possible  to  assess  the  planned  testing  for 
the  suitabili^  objectives.  In  addition,  critical  suitability  assets  may  not  be  available  when 
lequired  during  the  testing  period.  Such  factors  could  result  in  the  non-completion  of  testing  and 
increase  the  risk  of  making  inappropriate  decisions  at  the  decision  milestones. 

OUTLINE  FOR  REDUCING  RISK 

The  compatibility  of  the  schedule  with  the  program  decision  milestones  and  the  compatibility  of 
the  hardware  development  and  testing  widi  the  software  development  and  testing  also  should  be 
evaluated.  One  should  review  the  supplementary  schedules  and  data  source  matrix  (DSM)  in  the 
TEMP,  as  well  as  the  timing  sequences  relating  software  test  to  hardware  testing.  The  integrated 
schedule  should  be  examined  to  determine  if  the  time  and  resources  that  are  available  for  the 
suitability  portion  of  the  OT&E  are  adequate. 

a.  Schedule  Supports  the  Program  Milestones. 

Has  sufficient  time  been  scheduled  to  ensure  the  collection  of  meaningful  suitability-related 
data?  Will  results  be  available  prior  to  the  requisite  milestone? 

The  integrated  schedule  should  demonstrate  that  the  testing  that  will  yield  suitability  results  at  a 
time  consistent  with  the  need  for  information  at  the  key  program  decision  milestones.  The  rela¬ 
tionship  of  the  test  periods  to  the  program  schedule  should  give  an  idea  of  the  amount  of  test 
time  that  will  be  available  to  supoort  the  various  milestone  decisions  and  the  associated  reporting 
requirements.  (The  details  for  suitability  testing  are  discussed  in  sections  2.4  and  2.4.4.) 


Th«  system  lOT&E  consists  of  750  flight  hours  and  will  be  completed  at  least  120 
days  prior  to  Milestone  III;  thus,  there  should  be  many  opportunities  for 
mainfepance  actions  The  test  phase  will  provide  adequate  maintenance  data  to 
present  the  suitability  results  to  the  Milestone  III  DAB. 
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b.  Scheduling  Suitability  Elements. 

Does  the  schedule  include  an  on*dock  date  for  support  equipment?  Will  maintenance 
personnel  training  be  completed  in  time  to  support  OT&E  maintenance-related  testing? 

The  integrated  schedule  should  give  an  indication  of  when  the  various  suitability  elements  will 
be  available  for  OT&E;  this  includes  when  the  support  equipment  will  become  available  for  the 
various  levels  of  support,  when  representative  maintenance  personnel  will  become  available  for 
the  vanous  levels  of  support,  etc.  The  placement  of  these  items  on  the  inte^ted  schedule  will 
give  the  reviewer  an  id^  of  when  various  pomons  of  the  operational  suitability  evaluation  will 
be  completed  and  when  the  results  of  this  activity  will  become  available  for  presentation  to  the 
decisionmakers. 


For  the  lOT&E,  the  first-  and  second-level  maintenance  of  the  system  will  be 
performed  by  representative  military  personnel.  Prototype  test  and  support 
equipment  will  be  available  for  the  test  phase.  This  phase  will  provide  the  first 
operational  testing  of  the  test  equipment 


c.  Scheduling  Adequate  Time. 

Is  adequate  time  scheduled  for  the  suitability  portions  of  operational  testing? 

One  should  review  the  schedule  to  ensure  that  the  support  elements  required  to  meet  the 
suitability  objectives  have  been  identified;  this  includes  testing  and  associked  hardware  and 
facility  t^uiretnents  down  to  the  lowest  practical  level.  If  program  schedules  have  slipped,  then 
resch^uling  of  reviews  should  be  examined  to  ensure  that  ad^uate  time  for  OT&E  conduct  is 
plaimed  prior  to  the  scheduled  review. 


The  750  tlighi  hours  that  are  planned  for  lOT&E  will  provide  a  sufficient  number 
of  the  two  designated  system  missions  to  demonstrate  the  system's  reliability  for 
the  missions.  Scheduled  "on-dock*  dates  for  support  equipment,  spares,  T.O  s, 
and  training  are  supportive  of  the  planned  start  of  flight  testing 


d.  Software. 

Are  adequate  time  and  necessary  resources  scheduled  for  the  planned  operational  testing  to 
allow  for  software  testing  concurrent  with  hardware  testing?  Will  the  software  be 
baselined  and  under  configuration  control  prior  to  the  start  of  OT&E? 

One  must  ensure  that  the  TEMP  includes  key  software  releases,  subsystem/system  tests,  and 
sufficient  key  system  events  to  coordinate  software  testing  with  the  system  schedule.  The  cal¬ 
ender  time  available  for  software  testing  should  be  indicated.  Resources  necessary  to  support 
software  testing  must  be  planned  for  and  scheduled. 


The  integrated  schedule  provides  visibility  of  the  schedule  for  planned  soflware 
block  changes  and  clearly  indicates  a  logical  time-phased  sequence  for  their 
introduction  into  the  OT&E  process  in  a  controiied  and  organized  fashion. 
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The  purpose  of  the  Management  section  Is  to  outline  the  T&IB  responsibilities  of 
primary  participating  organizations  (developers,  testers,  evaluators,  and  users). 
The  T&E  strategy  should  be  related  to  the  acquisition  stnttegy  of  the  program 
(any  concurrency  of  production  and  testing  should  be  discussed).  The  key  deci¬ 
sion  points  should  be  listed,  along  with  the  T&E  reports  that  will  support  those 
decisions.  Terms  such  as  “Low  Rate  Initial  Production,”  "Fu!l  Rate  Production,” 
and  “Initial  Operational  Capability”  should  be  defined  quantitabvely  (rate  and  total 
quantity).  The  scheduled  date  (e  g.,  fiscal  year  quarter)  for  the  decision  to  pro¬ 
ceed  beyond  LRtP  should  be  identified.  The  management  of  schedule,  resource, 
or  budgeting  constraints  that  may  Impact  the  adequacy  of  planned  testing  should 
be  addressed. 

AREA  OF  RISK 

Inadequate  emphasis  may  be  placed  on  addressing  the  suitability  issues  of  the  OT&E. 

The  risk  to  operational  suitability  in  the  T&E  management  areas  is  that  the  management  stnicture 
of  the  program  may  place  inadequate  emphasis  on  the  suitability  issues  of  the  T&E.  Some  man¬ 
agers  tend  to  focus  attention  on  items  that  are  important  to  evMuating  operational  effectiveness 
and  pay  little  attention  to  the  operational  suitability  objectives.  This  management  emphasis  is 
the  result  of  the  chronological  sequence  that  places  effectiveness  before  suitability. 

OUTLINE  FOR  REDUCING  RISK 

Reducing  of  risk  requires  the  manager  to  be  held  accountable  for  both  effectiveness  and 
suitability  objectives  over  the  life  of  the  test  Accordingly,  these  responsibilities  are  to  be  clearly 
identified  in  the  TEMP.  Resources  and  schedules  should  provide  Ae  capability  to  perform  the 
required  testing. 

a.  Management  Responsibilities. 

Is  there  an  outline  of  management  responsibilities  to  ensure  suitability  objectives  receive 
the  proper  attention? 

The  responsibilities  that  are  spelled  out  should  give  the  suitability  objectives  the  needed 
attention.  The  proposed  management  team  and  approach  will  indicate  the  organization  that  will 
be  conducting  and  contributing  to  the  T&E  in  the  suitabiliiy  area.  In  some  cases,  suitability 
elements  may  be  evaluated  by  supponing  organizations.  These  organizations  need  to  be 
identified  in  early  versions  of  the  TEMP  to  ensure  that  resources  are  available  when  required. 


The  Test  Integration  Working  Group  (TIWG)  will  interlace  with  the  following 
functional  groups:  the  Integrated  Logistics  Support  Management  Team  (ILSMT), 
the  Training  Support  Working  Group,  the  MANPRINT  Joint  Working  Group,  the 
Computer  Resources  Working  Group,  and  the  Operational  Test  Readiness  Review 
Working  Group. 
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b.  Proposed  Testing. 


Is  it  dear  that  testing  is  adequate  in  scope  to  provide  confidei'ce  in  the  planned  suitability 
results? 

A  sufficient  number  of  test  events  should  be  scheduled  to  provide  a  level  of  ccnffidence  in  the 
resulting  suitability  statistics.  If  significant  levels  of  risk  ate  the  result  of  limited  test  assets,  then 
these  ri^  should  be  discussed. 


The  lOT&E,  as  currently  estimated,  consists  ot  ground  tests,  jettison/separation 
tests,  captive  flights  and  the  launch/ftight  testing  of  17  missiles  using  the  F-18 
as  the  carrier  aircraft. 


c.  Schedule  Compatibility. 

Is  the  proposed  relationship  between  the  decision  milestones  and  the  T&E  reports 
compatible? 

The  management  section  of  the  TEMP  should  indicate  the  planned  sequencing  of  T&E  reports 
from  testing  phases  that  support  the  major  ;mgram  milestones.  The  definitions  of  the  program 
milestoiies  provide  a  context  for  assessing  the  adequacy  of  the  suitability  information  that 
will  be  available  at  that  milestone  decision  pcnnt 


Part  m  of  a  TEMP  Is  devoted  to  the  Development  Testing  and 
Evaluation  of  a  weapon  system  program.  Resporalbllltles  for  Ha 
content  testa  wHh  Deputy  Director,  Defense  Research  and 
En^naerlttg  (TSE).  JMa  section  almuld  summarize  those  aeOvtUea 
plumed  tor  the  development  test  phase(s).  Knowledge  of  those 
acthritlea  can  be  of  value  to  ttte  DOTiE  Staff  Assistant  In 
undersOmdlng  the  overall  testing  concepts  and  potential  availability 
of  data  to  accomplish  early  operational  assessments. 
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Part  IV  of  the  TEMP  focuses  on  the  Operational  Test  and  Evaluation  portion  of  the 
overall  test  program.  It  highlights  the  Critical  Operational  Issues  (COIs), 
summarize  the  OTiE  that  has  been  performed  to  date,  and  describe  the  OT&E 
that  is  planned. 

AREA  OF  RISK 

An  ineffective  test  program  may  be  stnicbiretL 

The  OT&E  description  may  lack  detailed  infcsmation  or  communicate  information  that  is 
unclear  and  ambiguous.  CMtical  Operational  Issues  (COIs)  may  not  be  identified.  Problems  or 
limitations  with  past  OT&E  may  not  be  included  in  the  OT&E  summary.  Limited  definidon  of 
the  planned  OTA£  may  result  in  the  T&E  "contract"  being  assumed  to  be  one  thing,  while  the 
plan  is  for  something  else. 

OUTLINE  FOR  REDUCING  RISK 

a.  OT&E  Overview,  (see  2.4.1) 

Is  the  OT&E  Overview  complete? 

This  section  should  provide  a  summary  of  how  the  OT&E  is  snuctuted.  It  should  show  how  the 
program  structure,  test  managemem  structure,  and  the  required  resources  are  telared  to  the 
system  requirements,  COIs,  test  objectives  and  decision  milestones.  It  also  should  show  how  the 
completed  OT&E  has  evaluated  the  system  and  how  the  future  OT&E  will  evaluate  the  systeitL 
This  overview  should  give  adequate  attention  to  the  suitability  COIs  and  discuss  hw  the 
suitability  evaluations/assessments  will  be  provided  for  each  of  the  decision  milestones. 


A  review  of  the  TEMP  indicated  that  the  OT&E  overview  was  an  attempt  by  the 
system  developer  to  justify  performing  a  scaled-down  OT&E. 


b.  Critical  Operational  Issues,  (see  2.4.2) 

Are  all  COIs  identified? 

Factors  which  could  preclude  suitability  performance,  as  required  by  the  users,  must  be 
identified.  Identification  and  description  of  the  related  COIs  are  required  to  focus  test  resources 
and  the  attention  of  the  decisionmakers  on  these  important  issues.  Progt^  aspects  that  may 
result  in  suitability-related  COIs  include  a  statement  of  higbCT  levels  of  suitabili^  perfonuance 
titan  that  requited  in  previous  similar  systems,  as  well  as  the  introduction  of  new  or  unproven 
technology  into  a  system. 
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Can  the  units  that  are  equipped  with  the  system  achieve  their  peacetime  and 
wartime  system  readiness  objectives  (SROs)?  The  system  shall  demonstrate  a 
peacetime  operationai  avaiiability  of  0  86  and  a  wartime  operational  availability 
of  0.78.  The  probabiiity  of  successfully  completing  a  three-hour  mission  shall 
be  0.70,  with  a  mean  time  between  operational  mission  failure  of  8  5  hours 


c.  OT&E  to  Date,  (see  2.43) 

Is  the  summary  of  OT&E  to  date  complete  and  accurate? 

The  description  of  prior  OT&E  should  not  be  a  duplication  of  the  detailed  OT&E  leports.  The 
discussion  should  summarize  the  prior  operational  tests,  including  what  portions  of  the  support 
system  were  tested  and  what  the  results  were;  what  suitability  COIs  have  been  fully  or  partly 
addressed?;  what  were  the  results?;  what  planned  activities  were  not  petformed,  and  why? 


The  lOT&E  was  initiated  in  January  of  1985  and  was  conducted  for  a  two-month 
period.  Insufficient  data  were  accumulated  to  provide  quantitative  measures  lor 
reliability  or  maintainability.  Five  qualitative  maintainability  deficiencies  were 
documented  in  the  report  for  this  short  test  phase. 


<L  Futtu«  OT&E.  (see  2.4.4) 

Does  the  discussion  of  ftiture  OT&E  indude  a  complete  description  of  each  OT&E  test 
phase? 

All  future  OT&E  should  be  described  and  its  specific  ptcpose  identified.  Any  major  deficiencies 
should  be  atldressed,  as  well  as  when  their  correction  be  verified.  E%h  phase  of  OT&E 
should  be  discussed  separately.  The  configuration  to  be  tested  in  each  phase  of  OT&E  should  be 
identified.  The  portion  of  the  support  system  that  is  present  in  each  (tf  these  test  phases  should 
be  discussed.  The  suitability  objectives  for  each  phase  of  OT&E  should  be  listed.  A  brief  ile- 
scription  should  portray  how  each  phase  of  testing  will  be  conducted  (events  to  be  poformed, 
types  of  representative  support  personnel  to  be  us^  how  the  system  maintenance  a^  logistics 
support  will  be  evaluated  in  this  phase,  and  the  role  of  suitability  noodeling  and  simulation  in  this 
ph^).  Those  factors  that  limit  the  full  and  completely  realistic  rrperatiorid  test  of  the  suitability 
aspects  of  the  system  should  be  identified. 


The  lOT&E  supports  Milestone  III  and  will  consist  of  a  60-day,  300-flying-hour 
effort  using  three  prototype  aircraft.  The  test  will  be  conducted  between  July  and 
October  1990.  Logistics  and  maintainability  demonstrations  will  be  conducted  to 
provide  an  early  assessment  of  the  user's  ability  to  maintain  the  system  under  a 
concept  of  two  levels  of  maintenance. 
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The  OTiB  Overview  should  provide  a  summary  of  how  the  OT&E  Is  structured.  It 
should  show  how  the  program  structure,  test  ttnaragemerd  structure,  and  the 
required  resources  are  related  to  the  system  requirements,  suitability  COIs,  test 
ot^ecthres  and  decision  milestones.  It  also  should  show  how  the  completed 
OTiB  has  evaluated  the  system  suitability  and  how  the  future  OTiB  will  evaluate 
the  system.  This  overview  should  give  adequate  attention  to  the  suitability  COIs 
and  tOscuss  how  the  suitability  evaluations/assessments  will  be  provided  for  use 
In  each  of  the  decision  milestones. 

AREA  OF  RISK 

An  inadequate  description  leads  to  the  approval  of  an  unacceptable  suitability  test  and 
evaluation  program. 

If  the  discussion  in  the  OT&E  outline  is  inadequate,  or  if  the  coverage  of  tqterational  suitability 
is  not  mirqiiatr,  then  the  information  that  is  us^  to  judge  the  accept^ility  aS  die  TEMP  and  the 
test  program  alM  will  be  deficient  As  a  result  the  test  program  that  is  approved  may  be  unable 
to  meet  the  needs  of  the  decisionmakers. 

OUTLINE  FOR  REDUCING  RISK 

To  reduce  the  risk  associated  with  inadequate  testing,  it  is  important  tq  have  an  understanding  of 
the  testing  already  conducted  and  that  which  is  planned.  The  OT&E  Overview  should  include  the 
contractor  testing  and  early  technical  testing  as  well  as  eariy  operational  lest  and  evaluation.  The 
operational  evaluation  must  take  advantage  of  testing  data  fit^  all  appropriate  sources  up  to  and 
including  the  independent  operational  test  and  evaluation.  Irollow  on  testing  should  be  used  as 
soon  as  it  is  performed  to  assist  in  the  evaluation  of  the  production  articles.  The  Overview 
indicates  the  level  of  development  of  test  articles  being  used  in  the  operational  test;  for  example, 
LRD*  or  prototypes.  If  the  test  articles  will  not  be  full-rate  production  articles,  there  should  be  a 
discussioa  of  the  differences  between  the  test  and  full-rate  production  articles  and  the  effects  of 
tbeir  use  during  OT&E. 

a.  OT&E  Overview. 

Is  the  OT&E  Overview  in  the  TEMP  a  complete  saimnary  of  the  suitability  T&E  program? 

The  OT&E  Overview  should  provide  a  clear  understanding  of  the  testing  already  completed,  the 
testing  yet  to  be  completed,  and  who  win  process  and  evaluate  the  data.  The  Overview  should 
be  or^nized  by  acquisition  phase  and  specific  information  requirements;  for  example, 
information  to  the  system’s  developer  should  be  annotated.  Special  suitability  testing 
requiremenrs  should  be  discussed;  for  example,  the  impact  on  the  test  of  an  inadequate  test 
support  package,  an  independent  contractor  to  perform  IV&V  on  the  system’s  software,  or  the 
use  of  the  developing  contractor  to  operate  or  maintain  the  tested  system  during  the  lOT&E. 
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The  pre-Milestone  II  TEMP  for  an  aircraft  indicated  in  the  OT&c  Overview  that 
an  effort  would  be  made  to  conduct  early  operational  capability  tests  to  provide 
the  user  with  a  perspective  of  the  potential  etfectiveness  anr*  suitability  during 
system’s  development  It  was  stated  that  OT  evaluators  will  participate  in 
contractor  and  technical  testing  exercises,  to  include  demos,  surveys,  and  mock- 
ups.  The  Overview  stated  that  an  independent  contractor  would  be  used  to  conduct 
IV&V,  and  this  contractor  would  be  available  to  the  OTA  to  assess  systems'  soft¬ 
ware  suitability  Because  operating  the  system  in  a  chemical  environment  is  a 
major  concern,  the  Overview  provided  a  discussion  of  the  operational  testing  of 
system's  maintenance  while  in  this  environment  The  Overview  also  stated  that 
the  lOT&E  would  be  conducted  using  LRIP  aircraft  (with  a  brief  description  of  the 
difference  between  LRIP  and  Full-Rate  production  systems),  and  that  resulting 
data  would  support  the  full-rate  production  decision  at  Milestone  III 


b.  OT&E  Summary. 

Does  the  DOT&E  Overview  summarize  the  OT&E  that  has  already  been  conducted,  to 
indude  test  artides  descriptions  and  the  future  OT &E? 

The  Overview  should  discuss  the  OT&E  conducted  to  date,  by  phase,  and  outline  the  successes 
and  failures  in  achieving  the  operational  suitability  characteristics.  If  reviews  and  decisions  were 
made  that  altered  future  system  development  or  affected  testing,  there  should  be  a  discussion  of 
how  these  issues  will  be  addressed.  The  Overview  should  provide  a  test  article  description  that 
states  the  level  of  development  of  the  tested  system,  and  a  synopsis  of  the  OT&E  events  with  a 
discussion  of  the  results  of  each  event  FinaUy,  the  Overview  should  provide  a  summation  of 
program  management  decisions  that  will  impaa  on  the  acquisition  schedule  or  operational 
testing  and  thereby  require  adjustments  in  test  resources  or  test  design  and  execution. 

The  future  OT&E  planned  for  the  system  must  be  applicable  for  the  phases  of  the  system’s 
acquisition  and  in  the  tinK  frames  required.  This  sermon  should  provide  a  test  article  description 
of  the  system  that  will  be  tested  in  each  of  the  future  phases  and  a  discussion  of  the  OT&E 
objectives  for  each  test  There  also  should  be  a  detail^  discussion  of  the  OT&E  events  and 
scope  of  the  testing,  as  well  as  a  discussion  of  the  basic  scenarios  that  will  be  followed  for  each 
of  die  tests. 


In  evaluating  the  TEMP  for  a  system,  it  was  discovered  that  the  future  OT&E 
section  stressed  the  completeness  of  past  testing  and  attempted  to  justify  a  lack  of 
planned  future  suitability  OT&E  for  the  system.  The  TEMP  stated  that  because 
Service  personnei  wouid  be  participating  in  the  DEM/VAL  phase,  have  access  to 
contractor  IV&V  data,  and  participate  in  demos,  iOT&E  should  be  reduced  in  scope 
to  a  30-day  field  exercise.  The  TEMP  proposed  concurrent  DT/OT  prior  to 
Milestone  III,  with  each  test  organization  having  equal  access  to  results  Rnally,. 
it  proposed  that  an  Early  Operational  Capability  Unit  participate  and  maintain  the 
system  in  lOT&E  because  this  would  provide  a  head-start  on  the  training  of  the 
Initial  Operational  Capability  Unit.  Many  cl  these  statements  tend  ‘3  dilute  the 
operational  test  by  reducing  the  test  time  and  resources,  and  by  providing  non¬ 
typical  user  troops. 
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The  Critical  Operational  Issues  (COIs)  are  key  operational  eftectiveness  and  oper¬ 
ational  suitability  Issues  that  must  be  examined  In  OTiE  to  determine  the 
system’s  capability  to  perform  Its  mission.  The  COIs  are  not  characteristics, 
parameters,  or  thresholds,  but  they  may  have  associated  characteristics, 
parameters,  or  thresholds.  COIs  should  cover  all  areas  that  critically  affect  the 
system's  ability  to  accon^riish  Its  mission  In  the  Intended  environment.  The 
TEMP  should  Identify  which  phase  of  OT&E  vrill  address  each  COI. 

AREA  OF  RISK 

Adequate  attentioa  may  not  be  focused  on  some  specific  point  that  is  important  to  the 
successfid  fielding  of  the  system. 

The  significance  of  the  COIs  is  that  they  help  focus  lesouices  and  management  attention  on 
items  Aat  aie  important  in  evaluating  the  system's  progress  toward  attaining  its  operational  ob¬ 
jectives.  The  COIs  allow  the  decisionmal^  to  have  a  smaller  set  of  issues  to  address  when 
making  decisions  on  the  acceptability  of  system  development  to  dale.  Within  this  context,  the 
risks  ate  that  the  COIs  may  be  impropetiy  identified  or  that  a  critical  issue  will  be  missed  and 
the  decisionmakers  will  not  focus  a^uate  attention  on  some  point  that  is  important  to  die 
successful  fielding  of  the  system. 

OUTLINE  FOR  REDUCING  RISK 

The  Critical  Operational  Issues  (COIs)  are  the  critical  aspects  of  t^e  system's  operational 
effectiveness  arid  operational  suitability  that  are  intended  for  examination  and  resolution  during 
OT&E  Critical  operational  issues  ate  developed  by  the  tester  and  may  be  represented  as  ques¬ 
tions  that  must  be  answered  at  the  next  acquisition  decision  milestone.  The  (X>Is  are  not 
characteristics,  parameters,  or  thresholds,  but  they  may  have  associated  characteristics, 
parameters,  or  diresholds.  The  issues  should  cover  ^  areas  that  critically  affect  the  system's 
ability  to  accomplish  its  mission  in  the  intended  environment.  The  TEMP  should  identify  which 
phase  of  OT&E  will  address  each  COI. 

The  emphasis  of  COIs  is  on  the  determination  of  the  attainment  of  certain  key  performance 
levels  and  on  surfacing  potential  problems  that  could  interfere  with  successful  mission 
accomplishmenL  Critical  operational  issues  may  change  firom  one  milestone  to  the  next  as  some 
are  resolved  and  new  ones  emerge  in  keeping  with  the  systems  development  status.  COIs  must 
be  structured  to  ensure  that  the  information  needs  of  the  acquisition  review  body  can  be 
addressed  fm  the  milestone  at  hand. 

Reducing  die  risk  associated  with  OT&E  CXTIs  requires  that  the  system's  requirements,  mission, 
and  operating  and  support  concepts  be  understood.  Factors  that  are  impmtant  critical  issues 
related  to  operadtMial  suitability  should  be  identified  and  documented  in  the  TEMP.  The  full 
range  of  the  intended  operational  environment  must  be  considered.  The  list  of  COIs  should  be 
thorough  enou^  to  ensure  that,  if  every  CX)I  is  resolved  favorably,  the  system  should  be 
operationally  suitable  when  employed  in  its  intended  environment  by  tj^cal  users. 
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a.  Completeness  of  List  of  C01& 

b  the  list  of  suitability  COb  complete? 

The  majen'  risk  in  this  area  is  the  situati(»  where  some  inqxRtant  suitabiiity  area  has  been  over- 
locAed  and  thus  not  identified  in  the  list  of  GOb.  As  a  result,  attention  is  not  given  to  this  area 
when  the  Operational  test  plan  is  prepared,  and  the  qteradonal  testing  pnvidcs  inadequate  data 
to  evaluate  the  condition  at  the  time  of  an  inqxxiant  milestone  decision.  The  review  of  the 
TEMP  should  focus  on  identifying  any  critical  suitability  area  that  is  not  included  in  the  list  of 
COb.  Suitability  risk  areas  may  be  iwntified  (com  the  areas  of  highest  risk,  or  from  the  areas 
that  are  of  the  highest  criticality  witfam  the  supput  plans  for  the  system. 


A  remotely  piloted  vehicle  (RPV)  test  program  was  planned  to  address  three 
primary  objectives,  which  were  labeled  as  'critical*  issues.  Other  issues 
including  survivability,  RAM,  training,  and  human  factors  were  to  be  addressed 
only  to  the  extent  that  they  affected  the  RPVs  ability  to  meet  the  criteria  of  the 
three  'crtticaP  issues.  Program  milestone  documentation  did  not  contain  explicit 
suitability  criteria.  The  approach  to  suitability  testing  was  to  observe  the 
ability  of  the  RPV  to  support  the  mission  under  sustained  combat  operations, 
noting  any  shortfalls  that  could  be  attributed  to  suitability  problems.  This 
approach  was  justified  by  arguing  that  only  'critical*  issues  required  explicit 
critena,  and  therefore  suitabilify  did  not  require  explicit  threshold  values. 


b.  Suitability  Requirements. 

Are  the  suitability  requiremenb  at  high  levcb  compared  to  previous  systems?  Are  thoy 
identified  as  COb? 

The  planned  leveb  of  suitability  performance  (particularly  reliability  and  maintainability)  may 
be  COb  if  the  system's  successfid  operation  b  dependent  on  achieving  a  markedly  higba  level 
of  reliability,  maintainability,  etc. 


The  reliability  requirement  for  the  new  system  is  significantly  higher  than  the 
existing  system  (failure  rate  is  one-third  of  the  previous  rate).  A  COI  has  been 
identified  as  'has  the  system  achieved  the  planned  level  of  reliability?' 

c.  Syston  Technology  Risk  Areas. 

b  new  or  unproven  technology  required  for  titis  system,  or  pbumed  for  use?  Are  there 
COb  on  these  technologies? 

The  use  of  advanced  technologies  in  a  system  may  introduce  risk  both  in  achieving  system 
performance  leveb  and  in  achieving  the  ct^ability  to  support  such  new  technology.  Arc  there 
relatively  unjnoven  technologies  in  the  system"!  Ate  these  technologies  uixlerstood  fiom  a 
reliability  stant^xnnt?  b  the  process  to  suppon  these  technologies  (maintenance  procedures,  test 
mediods,  maintenance  training,  human  factors,  etc.)  understood  or  demonstrated? 


A  new,  unproven  cooling  method  is  proposed  lor  the  IR  seeker  of  a  guided  missile. 
The  reliability  of  the  cooling  and  the  seeker  is  critical  to  the  missile's  operation. 
Therefore,  the  reliability  and  maintainability  of  the  seeker  and  its  cooling  is 
specified  as  a  COI. 
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Bach  completed  phase  o1  OTAE  should  be  summarized.  Descriptions  of  the 
hardware  and  software  actually  tested  should  be  provided.  Differences  between 
the  system  used  In  testing  and  the  configuration  expected  to  be  fielded  should  be 
highlighted,  and  potential  Impacts  to  suitability  resulting  from  the  differences 
should  be  discussed.  The  actuM  suitability  testing  that  occurred  should  be 
summarized.  Including  events,  scenarios,  resources  used,  test  limitations,  results 
achieved,  and  the  evaluations  conducted.  Planned  suitability  objectives  that 
vrere  not  met  should  be  highlighted  and  explained.  Status  on  the  resolution  of  all 
siUtablllty-related  Critical  Operational  Issues  should  be  discussed. 

AREA  OF  RISK 

The  suitability-related  requiremoits  that  remain  for  the  ftiture  OT&E  phases  may  be 
misjudged. 

Discussions  may  concentrate  mote  on  the  t^terational  effectiveness  requiiements  and  ignore  the 
suitability  requiiements.  In  some  cases,  the  information  may  be  condensed  to  such  a  degree  that 
there  is  very  little  infonnation  provided  to  form  a  basis  for  judging  the  suitability-related 
requirements  that  remain  for  the  foture  OT&E  phases.  If  numeious  phases  of  OT&E  have  been 
completed,  the  ^iproptiate  OT&E  reports  may  be  referenced  and  only  the  most  recent  or 
pertinent  OT&E  results  included  in  the  TEMP  discussion. 

OUTLINE  FOR  REDUCING  RISK 

Reducing  the  risk  associated  with  suitability  requires  that  the  system's  suitability-related 
requirements  be  tested  The  test  results  should  be  examined  to  ensure  the  objectives  of  availabil¬ 
ity.  compatibility,  transportability,  safety,  human  factors,  interop^bility.  reliability,  wartime 
usage  rates,  maintainability,  manpower,  training,  supportability,  logistics  suppoitability,  software 
sui^ortability,  and  documentation  have  been  met 

a.  Summary  of  Actual  Testing. 

Is  the  previous  testing  for  suitability  summarized? 

The  previous  testing  should  be  summarized  to  include  events,  scenarios,  resources  used,  test 
limitations,  evaluations  amducted,  and  results  achieved  Specific  OT&E  reports  that  contain  the 
detailed  results  should  be  identified  This  summary  should  address  the  suitability  areas 
adequately  and  should  not  be  a  summary  of  the  effectiveness  testing  to  date.  Suitability  elements 
that  are  included  in  each  of  the  test  phases  should  be  identified 


During  phase  lA  of  the  OT,  the  suitability  evaluation  did  not  address  the  direct 
level  of  maintenance  support  because  the  test  equipment  that  was  to  be  delivered 
to  the  test  site  was  not  available  in  time  to  meet  the  dates  that  were  mandated  by 
the  scheduled  range  times. 
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b.  System  Configuration. 


Is  the  configuration  of  tbe  test  systems  used  in  previous  phases  identified?  Were 
similarities  and  differences  vrith  production  or  later  test  articles  summarized? 

The  eaily  phases  of  the  testing  are  necessarily  performed  on  systems  that  are  not  of  the 
production  configuration.  NoneAeless,  the  data  from  these  test  pha%s  are  valuable.  The  key  is 
to  place  the  test  results  into  context  with  the  configuration  of  the  test  articles  and  the  realism  of 
the  test  environment  The  summary  of  the  prior  OT&F.  phases  should  document  these  factors. 


The  systems  used  in  the  Phase  I  lOTSE  did  not  have  the  diagnostics  software 
included  in  operational  software  for  the  mission  computer.  Therefore,  the 
diagnostics  objectives  were  not  evaluated. 


c.  Suitability  Objectives. 

Are  the  suitability  objectives  of  the  prior  OT&E  activity  described? 

Tbe  discussion  of  prior  acdvity  can  be  relatively  brief  or  can  include  references  to  previous 
reports.  Tbe  critic^  questions  are-what  suitability  objectives  were  planned  to  be  addressed  by 
the  imor  test  phases,  and  what  were  the  results?.  Is  there  a  discussion  of  significant  events,  test 
conations,  scenarios,  resources  used,  limitations,  and  results  as  related  to  the  system's  suita^ty 
requirements? 


Phase  t  lOT&E  was  conducted  on  two  of  the  initial  prototype  systems.  Reliability 
data  were  collected  during  the  test  phase.  Since  the  DT&E  test  team  provided  the 
system  maintenance,  no  maintainability  data  were  collected.  A  qualitative 
assessment  was  made  by  observing  the  maintenance  as  it  was  performed. 


d.  Suitability  Results. 

What  were  the  results  of  the  prior  OT&E  phases  in  tbe  area  of  operational  suitability? 

The  summary  of  the  ^vious  operational  testing  phases  should  be  summarized  and  provide 
insight  into  the  suitability  issues  that  have  been  arl^ssed  and  tbe  results  that  were  achieved.  A 
listing  of  the  CXlIs  (inclurling  the  suitability  COIs)  should  be  included.  This  listing  should 
indicate  which  of  the  COIs  were  resolved  (satisfactory,  unsatisfactory,  yes,  no,  etc.),  partially 
resoh'erl,  or  unresolved  at  tbe  completirw  rrf  that  phase  ol  testing. 


The  threshold  for  this  time  frame  was  established  as  a  system  reliability  of 
greater  than  0.80,  so  the  issue  was  satisfactory  for  the  Phase  I  OT&E.  (Since  the 
thresholds  will  change  for  the  various  phases  of  operational  testing,  this  success 
does  not  mean  that  the  suitability  will  automatically  remain  satisfactory.  The 
thresholds  will  become  more  demanding  in  the  iater  phases  of  OT.)  in  the  case  of 
the  maintainability  COi,  the  qualitative  maintainability  assessment  concluded  that 
the  access  provisions  for  the  required  preventative  maintenance  tasks  were 
unacceptable.  Since  the  design  of  the  access  provisions  were  not  projected  to  be 
revised  in  later  development  configurations,  this  deficiency  was  highlighted  as  a 
potential  limitation  on  the  system's  operational  suitability  once  it  was  fielded. 
The  proi  'm  manger  was  directed  to  resolve  the  deficiency. 
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All  remaining  OT&E  required  to  resolve  suitability-related  Critical  Operational 
Issues  should  be  discussed.  Operational  testing  should  be  described  to  verify 
the  correction  of  major  suitability  deficiencies  that  ware  previously  Identified.  A 
major  deficiency  Is  one  that  precludes  the  system  from  being  designated  as  “op¬ 
erationally  suitable.  “ 

AREA  OF  RISK 

Testing  nay  be  structured  so  that  a  critical  suitability  issue  will  be  missed. 

Future  testing  may  not  be  adequately  described  or  the  OT&E  may  not  be  directed  at  some  key 
issue  that  must  be  examined  befme  the  apprt^niate  decision  milestone.  The  results  of  earlier 
testing  may  identify  the  need  for  testing  of  (teficient  areas  that  have  since  been  collected. 

OUTLINE  FOR  REDUCING  RISK 

Suitability  objectives  for  each  critical  issue  must  be  reviewed.  For  the  objectives  not  met  to  date, 
one  must  ensure  that  the  future  test  program  will  adequately  address  the  issues  prior  to  the  appro¬ 
priate  milestone.  Those  deficiency  collections  dealing  with  suitability  issues  should  be 
identified  and  testing  planned  to  verify  the  cmiection.  Each  opmtional  test  should  be  described 
by  the  hypothesis  being  tested,  the  system  configuration  diat  will  be  tested,  the  scenario(s),  and 
the  sample  size  of  die  test  (e.g.,  repetitions,  hours,  etc.). 

a.  OT&E  Objectives. 

Are  the  OT&E  objectives  in  the  area  of  suitability  listed  in  sufficient  detail  to  be  addressed 
during  the  next  pluise  of  testing? 

This  section  should  summarize  the  objectives  that  will  be  the  focus  of  the  future  phases  of 
OT&E.  The  discussion  should  be  a  summary  of  the  detail  that  will  be  found  in  the  OT&E  test 
plan.  However,  it  most  provide  enough  detail  to  ensure  that  the  plaiuied  test  program  will  meet 
all  of  the  management  information  objectives.  Objectives  in  the  suitability  area  should  be 
summarized  to  ensure  tiiat  all  critical  areas  will  be  adrhessed. 


Objectives  included  evaluating  the  R&M  o<  the  RPV  system  in  its  operational 
environment  and  the  adequacy  of  the  planned  logistics  support  for  the  system 


b.  System  Configuration. 

Is  the  configuration  of  the  test  systems  to  be  used  identified?  Are  similarities  to  and 
differences  tetween  the  production  or  later  test  artides  summarized? 

The  key  to  placing  the  test  results  into  context  is  the  configuration  of  the  test  articles  and  the 
realism  of  the  test  environment  The  differences  between  the  planned  test  systems  and  the  pro- 
ducticHi  systems  should  be  clearly  identified. 
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The  OT  Phase  B  will  be  conducted  with  25  developmental  sets.  These  sets  will  ap¬ 
proximate  the  production  configuration,  except  for  the  computer  operating  sys¬ 
tem.  The  software  that  will  be  used  will  have  the  Block  II  operating  system  in¬ 
stead  of  the  Biock  III  planned  for  the  production  units. 


c.  OT&E  Events,  Scope  of  Testing,  and  Scenarios. 

Is  the  test  scenario  representative  of  the  actual  suppon  environment  and  are  suitability 
issues  addressed  for  planned  test  events? 

Evaluating  suitability  requires  that  the  system  be  performing  at  a  tempo  that  is  representative  of 
actual  operation,  and  that  the  support  ne^  posed  by  die  test  systems  bs  comparable  to  that  pro¬ 
jected  for  the  actual  operation.  The  sumn^  should  indicate  the  type  of  personnel  who  will 
maintain  the  system,  the  status  of  the  logistics  support  (e.g.,  unit  level  maintenance,  unit  and 
direct,  etc.),  the  maintenance  documentation  that  wUl  be  ust^  and  the  environment  under  which 
the  system  is  to  be  employed  and  supported  during  the  testing.  If  information  from  outside  of 
OT  be  used  by  the  OTA  to  supplement  the  data  fiom  this  OT  phase,  e.g.,  DT  data,  modeling 
and  simulations  (M&S),  etc.,  diese  sources  also  should  be  identified.  Any  plaiuied  use  of  M&S 
should  be  identified,  along  with  a  reference  to  the  M&S  verification  plan  or,  in  the  case  of  exist¬ 
ing  M&S,  an  explanation  of  when  and  by  whom  they  were  accredited. 


The  five  tesf  aircraft  wiii  be  flown  on  representative  missions  of  the  planned  du¬ 
ration.  The  systems  will  be  exposed  to  the  predicted  combat  operational  stress. 
The  sortie  rate  will  be  less  than  predicted,  (only  1  to  1.5  sorties  flown  each  day 
for  each  aircraft).  The  test  stations  that  wiil  be  available  at  the  test  site  are  the 
projected  support  level  for  a  24  UE  squadron;  therefore,  there  will  be  second 
level  maintenance  capability  that  is  in  excesii  of  that  planned  for  the  operational 
squadron.  The  evaluation  will  be  adjusted  for  (Ms  oifference. 


d.  Test  Limitatioiis. 

Are  factors  idoitified  that  may  predude  ftdl  and  complete  operational  testing 
involving/coacentrating  on  suitabiUty  issues? 

Limitations  listed  in  this  section  should  indicate  which  of  the  OT&E  objectives  and  which  GOls 
will  not  be  addressed  during  this  test  phase.  The  limitations  could  include  threat  realism, 
resource  availability,  limited  operational  environments  (military,  climatic,  etc.),  limited  support 
enviromnent,  maturity  of  the  test  system,  safety,  etc.,  that  will  preclude  a  full  and  cotrqtletely 
realistic  qietational  test  The  discussion  of  the  limitations  sh^d  address  the  impact  of  die 
limitations  on  the  resolution  of  the  affected  (Xlls  and  the  conclusions  regarding  the  operational 
effectiveness  and  opmtional  suitability  of  the  system.  The  COI(s)  diat  ate  affected  should  be 
identified  after  each  limitation. 


The  number  of  available  prototypes  will  limit  the  test  to  an  examination  of  the 
performance  of  a  single  combat  team.  The  interoperability  of  multipie  teams  will 
not  be  evaluated  during  this  test  period  (COI  S-6).  The  maintenance  test 
equipment  for  the  direct  levei  of  maintenance  support  will  not  be  available  in 
this  test  phase  and  the  mean  time  to  repair  at  the  direct  ievel  will  not  be 
evaluated  (COI  S-2).  The  test  site  for  this  phase  of  OT  does  rxit  include  the 
variation  in  topology  to  evaluate  the  transmission  capability  under  all  of  the 
terrain  conditions  specified  in  the  users'  need  statement  (COI  S-8). 
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A  summary  of  the  suitability-related  resources  to  be  used  during  the  test  program 
should  be  provided.  This  should  Include  major  range  and  unique  Instrumentation 
requirements  necessary  to  accomplish  the  OTAB  suUabllity-related  objectives. 
As  system  development  progresses,  test  resource  requirements  must  be  reas¬ 
sessed  and  subsequent  TEUP  updates  must  reflect  any  changes. 

AREA  OF  RISK 

Tinimg  and  quantities  of  resources  may  not  be  adequate  to  ensure  a  realistic  test. 

The  principal  resource  areas  of  operarional  suitability  risk  lie  in  the  lack  of  resources  to  provide 
adequate  test  rime  or  an  environment  that  has  insufiicient  realism.  With  the  wide  and  complex 
array  of  responsibilities,  test  planning  may  fail  to  focus  upon  some  of  the  (^terarional  suital^^ 
objectives  that  can  be  of  criri^  inqKirtance  in  the  toleration  of  the  system. 

OUTLINE  FOR  REDUCING  RISK 

Sufficient  time,  test  articles,  and  other  resources  must  be  scheduled  to  ensure  adequate  sample 
size  and  testing  of  the  suitability  characteristics  of  the  system.  Planned  test  aiticle  quantities,  test 
phase  duration,  and  odier  cridcd  suitability-related  test  resource  parameters  should  be  identified. 

a.  Resources. 

Are  the  tuimber,  timing,  and  conf ,^ration  of  hardware  and  software  test  articles 
spedhed?  Are  unique  and/or  modified  hardware  and  software  test  support  equipments 
identified?  Are  requirements  for  critica]  operating  force  support  and  spei^  reqmrements 
identified?  Are  test  support  spares  and  repair  parts  provided  for?  Have  system  simulation 
requirements  been  identified? 

The  number  and  timing  of  hardware-  and  software-required  test  articles  must  be  identified,  and 
be  adequate  to  provide  the  required  suitability  test  data.  Differences  between  test  and  production 
articles  should  be  defined,  with  probable  impact  on  suitability  test  results  identified.  supprat 
equipment  and  technical  infoimarion,  requi^  for  testing  m  each  phase  and  for  each  major  type 
of  OT&E,  should  be  identified.  Descriptions  should  include  any  appnqiriate  measure  of  test 
duration,  e.g.,  hours,  sorties,  etc. 

Unique  or  modified  suppon  equipment  should  be  identified  by  test  phase.  Equipment  rcquirL-ig 
spet^  calibrations  should  be  identified  by  source,  and  calibration  requirements  specified. 

The  oper^onal  fmee  support  requirements  should  be  identified  for  each  phase  for  testing,  e.g., 
aircraft  flight  hours,  ship  steaming  days,  T&E  units,  etc.  The  support  to  each  test  element  or  unit 
and  test  phase  shwld  be  adequate  tot  a  credible  rqierational  suitability  test  in  light  of  the 
operation  and  maintenance  concepts  planned  for  the  system. 
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Any  planned  operational  suitability  system  simulations,  including  computer-diiyen  simulation 
mo^ls  and  haidwaie-in-loop  test  beds,  should  be  defined.  The  system  simulation  requirements 
should  be  compared  with  existing  and  programmed  capabilities.  The  process  used  to  establish 
the  credibdity  of  the  tool  should  bt  idendfi^. 

Special  data  processing  equipment,  special  databases,  and  restricted/special  use  air/sea/land 
spaces  should  be  ider.dHed  and  specifi^  The  overall  test  data  gathering,  processing,  and  quali¬ 
ty  control  procedures  should  be  explained  and  should  be  adequate. 


One  prototype  system  (which  is  functionally  dentical  to  the  production 
configuration)  with  Block  I  soflware  will  be  dedicated  to  maintainability  testing 
during  Phase  I  A  production  model  with  Block  III  software  (which  incorporates 
automatic  equipment  reconfiguration)  will  be  dedicated  for  continued 
maintainability  testing  during  Phase  II  The  PROSTAR  model  will  uc  used  in 
conjunction  with  the  ALTSTAR  simulation  for  a  broader  assessment  of  logistics 
supportability  issues.  This  combination  of  model  and  simulation  has  been  veri¬ 
fied  and  validated  on  the  Air  Force  SEEK  ELF  program  using  actual  field  data. 
Required  test  support  spares  and  repair  parts  have  been  identified.  No  special 
equipment,  databases,  or  test  support  is  required  for  OT&E  suitability  testing. 


b.  Budgeting  and  Scheduling. 

Are  appropriation  line  numbers  provided  for  suitability-related  resoturces  and  identilied  by 
fiscal  year  and  program  element  numbers?  Are  need  dates  scheduled  for  key  suitability- 
related  test  resources? 

All  costs  for  testing  should  be  accurately  identified  by  program  element  All  imos,  services 
and/or  commodities  should  be  included.  Need  dates  feu'  key  suitability-related  test  lesources 
should  be  documented,  including  such  things  as  unique  instrumentation,  support  equipment 
simulators,  models,  and  test  beds. 


Funding  for  the  lOT&E  is  identified  as  follows. 

FY89  (M$)  FY90  (M$)  FY91  (M$) 

RCLR-1  (PE  64321)  3T0.0  610  5  240.3 

RCDR-1  (PE  64789)  132.24  436  2  670  0 

Note:  Funding  for  specialized  mai.itainabi!ity  testing  instrumentation  is  included 
in  PE  64321.  Associated  simulation,  modeling,  and  test  bed  suppmt  is  included  in 
PE  64789.  Reference  the  program  schedule  for  key  need  dates. 
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Chapter  3 
OT&E  PLAN 


Test  and  evaluation  plans  are  formal  planning  documents  that  provide  a  descriptitm  of  the  test  to 
be  conducted  and  tte  evaluation  muhods.  The  test  plan  provides  sufficient  details  about  die 
planned  test  to  assure  tiie  approval  autbority  that  test  ^jectives  will  be  addressed  satisfiicrorily. 
It  provides  guidance  to  the  Test  Director  with  regards  to  test  execution,  test  approach,  sanqtle 
sizes,  operational  environment,  how  the  threat  will  be  portray^  insttumentation  requirements, 
data  collection,  data  handling,  and  test  data  presentation  during  testing.  It  ^dso  provides  the 
measures  of  effectiveness  and  measures  of  performance,  as  well  as  the  conq>arisons  to  be  made. 
The  evaluation  plan,  which  may  be  a  sqtarate  document,  describes  data  ban^g  and  processing, 
and  evaluation  methods. 

The  operational  test  and  evaluation,  and  the  OT&E  plan,  support  bod)  development  and 
production  decisions  made  by  decisionmaking  autiiorities  during  tte  system  acquiation  proems. 
The  plan  details  the  extent  to  which  the  system's  issues  and  criteria  will  be  addressed  during 
individual  phases  of  operational  testing.  C^tetalional  testing  should  address  each  critical  opera¬ 
tional  issue  and  dieret^  support  the  evaluation  of  system  operational  effectiveness  and  smt^niity 
and  the  decisionmaking  pro^s. 

There  is  ix>  standard  Department  of  Defense  fonnat  for  the  OT&E  plan.  Each  Service  has 
developed  its  own  sqtpro^  to  organizing  the  information  tiuu  must  te  ctmtained  in  the  plan. 
Table  3-1  was  constructed  from  the  Seivices’  policies  on  operatitmal  test  and  evaluation;  it 
shows  how  the  structure  ct  this  operational  suitability  guidance  document  relates  to  die  major 
sections  of  the  respective  Sovices’  operational  test  and  ri^nation  plans. 

This  guidann  document  provides  a  process  for  reviewing  an  OT&E  plan  to  ensure  that  the  test 
program  will  provide  for  efficient  and  effective  snitiibility  test  and  evaluation.  Oritical 
Opaational  Issues  are  reviewed  to  identify  potential  plications  in  the  operational  suitability 
area.  Once  the  focus  of  die  suitability  effort  is  determined,  then  the  lest  objectives  and  the  sup¬ 
porting  measures  of  suitability  are  deMmined. 

As  rrruch  of  the  test  and  evaluation  plarming  activity  is  directed  at  identifying  and  organizing  die 
test  assets  (c.g.,  test  systems,  Kst  ranges,  suprting  personnel,  etc.)  into  planned  test  events  and 
sequences,  tto  area  rmist  be  thorou^y  reviewed.  The  structuring  of  the  test  activity  sixmld 
ensure  that  what  is  performed  during  testing  is  adequate;  diis  requires  realistic  testing  with  re¬ 
spect  to  the  planned  operational  environments  and  sufficient  test  data  to  give  confiifence  in  ^ 
results.  An  accurate  and  credible  evaluation  of  the  system’s  snitabili^  for  operational  use  then 
can  be  conducted. 
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The  templates  that  follow  address  suitability  considerations  for  each  of  the  major  content  areas 
inclu^  in  OT&E  plans.  These  templates  are  organised  into  the  following  sections: 

3.1  Description  of  Test  Articles 

3.2  Scope  of  Test 

3.3  Operational  Issues 

13.1  Test  Objectives 

3.3.2  Suitability  Parameters 

3.4  Test  Limitations 

3.5  Test  Conduct 

3.5.1  Test  Scenario 

3.5.2  Test  Hours 

3.6  Data  Management 

In  reviewing  the  OT&E  plans  from  the  various  Services,  the  overriding  objective  is  to  ensure  diat 
clear  linkage  is  provided  throughout  the  total  test  program.  Sinqtly  stmed: 

•  the  cqierational  issues  must  be  clearly  defined  and  testing  must 
focus  on  each  issue; 

•  test  limitations  must  be  considered  and  appropriate  adjustments 
made  to  offset  the  effect  of  the  limitations; 

•  testing  must  be  planned  for  conduct  in  a  controlled  fashion  to  ensure 
valid  and  accurate  data  are  generated;  and 

•  data  must  be  collected  in  sufficient  quantity  and  in  a  controlled 
fashion  to  provide  substance  foi  meaningful  evaluatitm. 

In  the  final  analysis,  all  aspects  of  the  test  program  must  link  together  to  provide  for  a  ciedibie 
and  defendable  evaluation  of  the  weapon  system’s  suitability  and  effectiveness. 
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The  OTiE  test  plan  coverage  of  Ute  test  articles  should  Include  discussion  of  the 
number  of  test  arttcles  and  any  significant  differences  between  the  test  articles 
and  the  production  system  to  be  fielded.  It  should  discuss  the  planned 
conhguratlon  and  Integration  of  the  test  units.  Test  articles  will  vary  In  their 
Integration  and  maturity  level  during  the  acquisition  process;  therefore,  testing 
the  suitability  of  these  systems  must  take  th^e  differences  Into  consideration. 

AREA  OF  RISK 

Inadequacies  in  test  articles  can  limit  the  ability  to  test  and  influence  test  results. 

The  planned  test  articles  may  not  be  ctm^letely  lepiesentadve  of  the  planned  pnxlucdon  aidcle. 
The  number  of  test  aidcles  planned  may  not  be  sufficient  to  adequately  test  die  ^tem  under  the 
intended  operating  envinnunents.  These  limitadons  inu‘4  be  identified  and  considered  in  the  test 
planning. 

OUTLINE  FOR  REDUCING  RISK 

Reducing  the  risk  associated  with  OT&E  testing  requires  that  die  plan  clearly  communicate  the 
test  assets  and  identified  limitadons  and  risk  areas  associated  with  these  test  ardcles. 

a.  Number  of  Test  Articles. 

Is  the  number  of  test  articles  discussed  and  justified? 

The  number  of  test  articles  is  most  often  a  compromise  between  need  and  cost  The  operational 
suitability  implications  of  the  test  articks  is  primatily  in  the  area  of  the  statistical  measures  diat 
requiie  a  number  of  trials  or  number  of  test  hours  to  reach  a  level  of  confidence  in  the  test 
te^ts.  The  number  of  test  articles  that  are  proposed  should  be  compatible  with  the  requirements 
for  confidence  in  the  measures.  The  kvel  of  reliability  to  be  demonstrated  can  be  a  majm  factor 
in  detetimning  the  number  of  test  articles  required. 


During  the  system  lOT&E,  only  live  receivers  will  be  available.  Three  will  be 
deployed  in  the  manpack  configuration  and  two  will  be  used  in  the  vehicular 
installation.  Operational  suitabiKty  data  will  be  collected  to  indicate  any  possible 
difference  in  reliability,  maintainability  or  logistics  supportability  of  the  two 
different  installations. 


Page  3-4 


OT&E  PLAN  -  Chapter  3 


b.  Configuration. 

Is  the  configuration  of  the  test  articles  comparable  to  the  planned  production 
configuration? 

The  cmifigurations  of  the  test  aiticles  and  the  production  systems  may  di^.  Differences  should 
be  review^  and  implications  in  the  operational  suitability  area  identified.  The  in^lications 
might  mean  that  die  systems  will  not  be  able  to  demonstrate  con^dbility  widi  other  items,  such 
as  test  equipment,  or  the  software  will  not  have  all  of  the  features  or  diagnostics  curability  that  is 
planned  for  die  p^uction  systems. 


The  test  plan  identifies  the  shortage  of  production  isolator  modules.  The  units  to 
be  used  for  the  Phase  IB  testing  will  not  have  isolator  modules  of  the  latest 
configuration.  Reliability  data  for  the  system  will  be  adjusted  to  reflect  the  use 
of  these  older  version  modules. 


c.  Alternative  System  Configuration. 

Does  the  test  address  an  planned  configurations  of  the  system? 

Some  s^tems  are  planned  for  cqieratiooal  use  in  a  number  of  different  configurations  or 
applications.  The  test  scenario  must  discuss  how  each  configuration  will  be  tested. 


Tt.d  system  configurations  consist  of  1 -channel,  2-channel,  and  5-channel 
navigation  signal  radio  receiver  sets.  The  Army  is  the  primary  tester  for  1-  and 
2<hannel  sets,  the  Air  Force  Is  the  primary  tester  for  the  5-channel  airborne 
set,  and  the  Navy  is  primary  tester  for  the  5-channel  shipboard  set.  Each 
Service's  scenarios  will  include  varying  mission  roles  and  mission  types  that 
could  impact  the  operational  suitability  of  the  various  configurations. 


d.  Suitability  Assets. 

Have  adequate  resources  been  provided  for  testing  of  the  suitalnlity  elements? 

The  evaluation  of  cqieratioaal  suitability  lequiies  Jiat  the  test  be  conducted  witii  adequate 
operational  suitabili^  assets  at  the  testing  site.  This  could  mean  test  equipment,  maintenance 
peisoi^l  and  facilities,  or  other  items  such  as  documentation.  A  limited  number  of  assets  could 
result  in  misleading  data  and  in^ipfopiiate  lesulls. 


The  second  level  of  maintenance  will  consist  of  test  equipment  with  the  capability 
to  verify  failures  and  to  isolate  the  failures  to  one  Shop  Replaceable  Unit  (SRU) 
95  percent  of  the  time.  (It  is  projected  that  5  percent  of  the  Line  Replaceable 
Units  (LRUs)  will  have  to  be  sent  to  the  depot  level  for  fault  isolation  and  re¬ 
pair.)  During  the  lOT&E  Phase  I,  only  one  intermediate  level  test  set  wrill  be 
available  at  the  test  site.  The  OTSE  Test  Director  (TD)  will  determine  which 
failed  test  units  will  be  troubleshot  and  repaired  at  the  test  site  and  which  are  to 
be  returned  to  the  contractor  for  factory-level  repair.  The  limitation  on  the 
number  of  test  sets  will  be  offset  by  the  TD’s  allocation  of  repairable  units.  He 
will  assure  that  a  cross  section  of  the  failed  units  are  tested  and  adequate  suit¬ 
ability  data  aie  gathered  at  the  test  site. 
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The  test  scope  will  provide  e  summary  of  the  relevant  Information  regarding  the 
number  of  systems  Involved  In  the  test,  test  location,  and  duration.  The  scope 
also  will  discuss  the  sequence  and  priority  of  the  test  phases  and  subtests. 

AREA  OF  RISK 

Limits  and  risks  inherent  in  the  planned  test  may  be  overlooked. 

The  scope  of  the  test  may  be  so  limited  that  die  system  will  not  be  exercised  sufficiendy  to 
demonsnrate  its  capability  to  meet  the  tqieiational  lequircments.  The  description  may  be  so  ab¬ 
breviated  that  the  reader  is  anable  to  jud^  the  limits  and  risks  inherent  in  the  planned  test 

OUTLINE  FOR  REDUCING  RISK 

The  plan  should  contiun  sufficient  descriptive  information  about  the  scope  of  the  test  to  indicate 
that  the  rest  scenario  that  is  planned  is  adequate,  that  the  test  environment  is  representative  of  the 
intended  cqietating  enviroameiit,  and  that  die  test  duration  will  allow  ope^onal  suitability 
elements  to  be  tal^  into  consideration.  Are  the  right  factors  and  conditions  included  to  ensure 
the  system  will  be  exercised  sufficiendy  to  capture  the  data  needed  to  answer  the  suitability  is¬ 
sues?  The  tactical  context  for  the  test  should  be  discussed  in  terms  of  type,  size  of  the  militaiy 
organizations  to  be  represented  or  simulated,  the  operations  to  be  conducted  relative  to  the  threat, 
intcropetability  elemrat  to  be  represented  or  simulated,  and  tbeir  conposite  relationship  to  the 
system  under  test 

a.  Test  Concept  amd  Scenarios. 

Is  there  sufficient  information  regarding  the  system's  different  configurations  and  concept 
of  anployment  during  the  testing? 

The  test  plan  should  inciede  sufficient  information  regaiding  the  employment  methods  plarmed 
for  die  test  and  the  use  of  any  different  conffgurations  of  the  system.  The  planned  scenarios 
should  be  conpared  to  die  doctrine  in  sufficient  detail  to  permit  examination  of  die  test  real- 
isriL  The  scope  of  the  plarmed  rest  should  be  described,  including  the  number  of  test  articles,  die 
anangement  of  the  test  assets,  die  numba  and  type  of  opo^uional  scenarios  that  will  be 
en^loyed,  the  manner  of  supporting  the  system  dumg  the  testing  ph^  and  the  range  and 
variations  of  test  environments  that  will  be  used.  Also  see  "Test  Strenario”  (see  section  3.S.l.a, 
p.3-18). 


The  system  requires  the  employment  of  three  major  elements;  a  master  station, 
user  units,  and  the  direct  support  team  vehicle.  Four  configurations  of  the  user 
units  will  be  employed:  manpack,  surface  vehicle,  auxiliary  ground  unit,  and 
airborne  (fixed  and  rotary  wing)  units.  The  test  site  configuration  employed  will 
consist  of  up  to  370  user  units  controlled  by  a  master  station  and  its  alternate. 
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b.  Support  System  Concept. 


Will  the  scope  of  testing  exercise  the  support  system  in  sufficient  detail  to  allow  evaluation 
of  suitability  issues? 

The  scenarios  and  test  events  should  include  events  that  will  trigger  the  use  of  suppoit  resources. 
The  scope  of  testing  should  include  the  use  of  the  suppon  structure  intended  to  support  die 
system  once  it  is  fielded.  Any  lunitations  within  this  area  should  be  highlighted. 


Five  levels  of  maintenance  are  included  in  the  planned  suppoit  concept.  The  test 
plan  describes  specific  maintenance  actions  to  be  performed  and  evaluated 
through  the  third  echelon.  Maintenance  manuals  will  be  evaluated  for 
completeness  and  consistency  with  the  planned  maintenance  skills  and  equipment. 


c.  Test  Environment 

Is  the  planned  test  environment  representative  of  the  environment  in  whidi  the  system  wUl 
be  operated  when  fielded? 

The  plan  for  employment  during  the  test  must  be  compared  to  the  intended  tqieratioiud 
environment  including  the  doctrine,  tactics,  and  threat  The  sccqie  of  testing  should  be  analyzed 
to  ensure  operational  suitability  issues  can  be  assessed 


The  test  will  involve  a  Marine  Amphibious  Bngade-size  unit  operating  in  rocky 
and  sandy  terrain  dunng  a  scheduled  field  exercise.  Weather  is  characterized  by 
warm  days  and  cold  nights,  with  possible  precipitation.  Special  events  will 
include  the  requirement  for  operating  in  an  NBC  environment,  wearing  gas  mask 
and  protective  gloves  Another  event  requires  cold  weather  testing  operating  the 
system  while  wearing  cold  weather  gloves,  mittens,  and  inserts. 


d  Interoperability  I^es. 

Are  the  interactions  between  the  system  under  test  and  other  systems  within  the  opoatiiig 
environment  consistent  with  assigned  missions? 

The  interface  requirements  and  operating  scenario  of  the  system  should  be  examined  to  ensure 
that  required  interfaces  are  included  in  tire  test  If  the  entire  range  of  interfiice  requirements  will 
not  be  tested  are  there  provisions  to  simulate  the  interfaces?  One  should  review  the  data 
requirements  to  ensure  that  information  concerning  interactions  among  systems  is  captured 


The  test  will  examine  the  ability  of  the  system  to  effectively  transfer,  receive, 
and/or  process  information  within  the  system  and  with  external  systems.  The 
test  plan  includes  a  special  imeroperability  test  using  a  specified  test  unit,  the 
PLRS  platoon,  and  other  personnel  equipped  with  the  EPLRS. 
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Operational  Issues  for  the  system  are  identified  In  the  TEMP.  The  list  of  the 
Critical  Operational  Issues  (COIs)  should  Identify  operational  suitability  features 
that  are  critical  to  mission  performance  and  the  ability  to  place  the  system  into 
field  use.  The  Issues  should  consider  the  total  system.  Including  critical 
subsystems  and  the  support  Hems,  the  system's  wartime  mission  requirements, 
and  Interfaces  wHh  other  systems  In  the  operating  environment.  Operational 
suitability  Issues  are  used  In  developing  the  test  objectives  or  test  issues. 

AREA  OF  RISK 

Major  issues  may  be  overlooked. 

Risks  associated  widi  the  suitability  issues  include  clarity,  coverage  of  all  missions  and 
scenarios,  and  coverage  of  the  plann^  operational  environment  The  suitability  CX>Is  may  be 
tuiclear  or  ambiguous  if  they  ate  not  described  ccunpletely.  There  is  risk  that  the  suitability  test¬ 
ing  will  be  focused  improperly.  Major  issues  may  be  overlooked.  Major  suitability  issues 
related  to  the  total  operating  environment  may  not  be  included. 

OUTLINE  FOR  REDUCING  RISK 

In  Older  to  reduce  risks  associated  with  operational  issues,  the  test  plan  must  have  a  thorough 
coverage  of  the  CXDIs;  this  includes  what  is  operationally  critical,  in  terms  of  the  system,  its  mis¬ 
sion,  requitetnents,  the  operating  environment,  and  the  supporting  oiganizations  and  structure. 

a.  Critical  Operational  Issues. 

Does  the  plan  address  the  COIs  that  were  identified  in  the  TEMP? 

The  OT&E  plan  for  the  system  should  address  the  COIs  identified  and  discussed  in  the  TEMP. 
The  COIs  should  be  relai^  to  a  particular  phase  or  phases  of  testing  for  resolution.  Does  the 
OT&E  plan  contain  test  objectives  to  address  these  COIs? 


The  system  concept  calls  for  two  levels  of  maintenance  for  the  elements  of  the 
radar.  This  is  identified  as  a  COi  in  the  TEMP.  The  test  plan  has  a  test  objective 
to  evaiuate  the  systems  reiiability  and  availability  as  it  relates  to  the  feasibility 
of  the  two-level  maintenance  concept. 


b.  Operational  Suitability  Issues. 

How  will  the  suitability  issues  faciUtate  answering  one  or  more  system  critical  issues?  Are 
the  suitability  issues  necessary? 

There  should  be  consistent^  between  the  critical  operational  issues  in  the  TEMP  and  the 
suitability  issues  and  objectives  addressed  in  the  OT&E  plan.  The  issues  should  contribute 
added  focus  on  approaches  to  assessing  the  operational  suitability.  Specific  test  events  should 
address  each  issue  and  provide  data  to  determine  whether  the  system  has  satisfied  each  issue. 
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The  system  OT-III  has  the  RAM  operational  issue,  ‘What  is  the  reliability, 
availability,  and  maintainability  of  the  user  equipment?  The  system  must  be 
sufficiently  available  to  support  the  basic  mission.'  To  support  this  suitability 
issue,  eieven  OT&E  test  objectives  were  developed  for  assessment  of  RAM 


c.  Operatioiial  Issue  DevdopmenL 

Are  the  operational  issues  devdoped  sufficiently  to  identify  the  operational  suitability  areas 
that  should  be  addressed  by  operational  testing? 

The  icquiiements  documentation  and  enqtloyment  doctrine  should  identify  the  elements  of 
system  peifotmance  that  could  ittqtact  the  suit^ility  of  the  system.  From  the%  descriptions,  the 
suitability  COIs  should  be  identified.  Tte  test  planning  should  relate  to  each  CX)I  and  the 
raticHiale  for  selecting  that  particular  issue.  Each  issue  sho^  be  a  focus  for  the  test  planning. 


The  test  plan  stales  ‘Can  user  equipment  be  effectively  integrated  into  a  wide 
range  of  weapons'  platforms  and  function  effectively  in  the  operational 
environmeiits  of  those  platforms?’  The  rationale  for  selecting  this  issue  states. 
This  COI  reflects  the  versatility  necessary  for  the  equipment  to  meet  the  unique 
requirements  of  each  of  the  Services  involved  in  this  program.’  It  is  apparent 
that  there  is  an  operational  suitability  issue  in  the  area  of  interoperability. 


d.  Suitability  Parameters,  (see  3  J 

Are  parameters  defined  for  each  of  the  suitability  COIs? 

To  determine  that  suitability  COIs  have  been  satisfied  lequiies  the  identification  of  parametet.^ 
that  can  be  measuied  and  will  provide  the  insight  lequiied  to  resolve  the  COIs.  The  four 
Operational  Test  and  Evaluation  Agencies  have  published  a  memorandum  of  agreement  on 
reliability,  availabilitj^and  maintaimdnlity  parameters  that  will  be  used  in  multi-Semcc  OT&E. 
This  list  of  panuneters  is  an  excellent  staning  point  for  identifying  the  parameters  diat  need  to  be 
examiiied  to  satisfy  the  suitability  COIs. 


The  suitability  parameters  to  be  used  during  the  test  will  be  as  follows. 

Reliability 

Mean  Time  Between  Operational  Mission  Failure 

Mean  Time  Between  Unscheduled  Maintenance 

Maintainability 

Maximum  Time  to  Repair  (90  percentile) 

Availability 

Operational  Availability 

Diagnostics 

ProbabiRty  of  Correct  Detection 

Mean  Time  to  Fault  Locate 
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Test  objectives  provide  an  overview  of  what  will  be  tested  during  a  particular  OT 
phase,  as  well  as  Identifying  the  Information  required  to  evaluate  whether  a 
specific  characteristic  of  the  sysmm  n)eets  the  requirements.  For  aacft  critical 
suitability  Issue  there  should  be  an  objective  which  supports  the  declslonnuklng 
process.  For  each  test  or  test  phase,  the  objective  should  be  supported  by  a  well 
understood  test  hypothesis.  The  hypothesis  may  not  necessarily  be  a  "statistical 
hypothesis."  It  should  focus  on  the  decision  that  arill  be  made  as  a  consequence 
of  the  test  results.  If  the  test  does  not  Influence  a  decision,  but  only  provides 
useful  data  or  Information,  then  It  Is  more  properly  termed  an  experiment 

AREA  OF  RISK 

Faulty  evahiatkm  criteria  will  lead  to  faulty  assessmoats. 

Test  objectives  may  be  pooriy  developed  or  ambiguoDS.  As  a  result,  the  evaluation  criteria  may 
not  be  properly  defined,  which  could  result  in  the  inability  to  adequately  evaluate  test  results. 

OUTLINE  FOR  REDUCING  RISK 

Suitability  test  objectives  should  be  well  defined  and  desc^tive  of  what  is  to  be  tested,  as  well 
as  what  data  ate  needed  to  assess  whether  or  not  the  objective  is  met  Objectives  should  be 
traceable  to  the  COI  or  other  issue  that  the  objective  supports.  Objectives  should  be  developed 
for  each  CX3I,  and  based  on  operational  requirements  or  some  oUict  quantitative  or  qualitative 
measure. 

For  each  test  or  test  phase,  there  should  be  a  description  that  includes: 

•  a  well-defined  test  objective  (or  test  hypothesis), 

•  the  sample  size  planned  (test  hours,  repetitions,  etc.), 

•  the  scenario  planned  for  the  test, 

•  the  number  of  test  articles, 

•  configuration  of  test  articles,  and 

•  what  is  imssing  from  the  system  car  what  the  difierences  are  between 
the  test  articles  and  tiie  planned  operational  configuration. 

This  description  could  be  presented  in  a  summary  matrix. 
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a.  Objectives  Consistent  with  Test  Rationale. 

Are  the  suitability  objectives  clearly  defined? 

Each  suitability  objective  should  be  clearly  defined  and  support  a  critical  operational  issue.  The 
objecnve  and  scope  should  be  consistent,  with  the  objective  being  more  specific  about  what  op¬ 
erational  charactensnc  must  be  assessed.  The  hypothesis  for  each  test  should  be  clear  firwn  the 
test  plan  discussion  How  will  the  test  provide  the  data  needed  to  prove  or  disprove  the  hypothe¬ 
sis?  The  measure  of  effectiveness  to  be  used  in  evaluating  the  suitability  objective  also  should 
be  stated. 


An  objective  states  ‘Evaluate  the  capability  to  maintain  the  1 -channel  manpack 
at  the  organizational  and  intermediate  levels  ’  Organizational  maintenance  will 
be  limited  to  buill-in-test  fault  detection/isolation,  and  battery  and  antenna  re¬ 
moval  and  replacement  At  the  intermediate  maintenance  level,  there  will  be 
technicians  and  test  equipment  to  perform  l-level  maintenance.  Mean  reptiir 
time  will  be  the  measure  of  effectiveness  for  on-  and  off-equipment  maintenance. 
Mean  repair  time  is  the  average  clock  hours  required  to  return  the  system  to  a 
serviceable  condition  excluding  administrative  and  logistics  delay  times. 


b.  Objective  Traceability. 

Does  each  objective  relate  to  a  critical  operational  issue? 

It  is  necessary  to  review  each  objective  and  compare  these  objectives  to  the  list  of  critical  opera- 
nonal  issuc(s)  to  ensure  that  each  appropriate  COI  is  addressed.  This  review  should  result  in 
each  objecnve  being  traceable  to  a  enneal  operational  issue  in  support  of  the  decisionmaking 
process. 


The  mean  repair  time  itiea«ure  ot  ettectiveness  (MOE)  is  traceable  through  the 
reliability  and  maintainability  oojechves  and  ''inally)  to  the  reliability, 
availability,  and  maintainability  critical  operational  .osue. 


c  Data  Requirements. 

Does  the  objective  describe  what  data  will  be  collected? 

Data  elements  necessary  to  evaluate  the  objecnve  must  be  identified.  The  information  may  be 
collected  either  by  test  representatives  or  as  a  result  of  the  normal  couise  of  events.  How  the  in- 
formanon  will  be  used,  as  well  as  any  necessary  adjustments  to  the  data,  should  be  discussed. 
(This  discussion  may  be  integrated  into  the  OT&E  plan,  in  an  appendix,  or  in  a  separate  Data 
Management  Plan.) 


The  mean  time  to  repair  includes  time  to  access  equipment,  troubleshoot,  repair, 
and  check-out  Data  elements  required  to  be  documented  include  maintenance 
actions,  task  times  required  to  pertorm  the  repair  and/or  servicing  of  the  user 
equipment  receivers,  the  contnbutions  ot  receiver  built-in  test,  and  intermedi¬ 
ate  test  set  capaoility  to  diagnose  correctly  the  unit  under  test.  The  test  team  will 
prepare  these  data  for  input  into  the  R&M  database 
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The  suitability  parameters  are  standards  by  which  operational  suitability  can  be 
gauged  and  evaluated.  They  may  be  quantitative  or  qualitative,  and  measure  a 
system's  performance  or  a  characteristic  of  the  system  that  Indicates  how  well 
the  system  performs  or  meets  a  requirement.  Depending  or  the  operational  suit- 
ability  element  being  examined,  there  may  be  multiple  suitability  parameters  that 
will  support  the  decisionmaking  process,  especially  when  the  suitability  element 
Is  Comdex  (e.g.,  reliability  or  maintainability).  In  addition,  suitability  parametars 
used  in  OTtE  should  be  representative  of  parameters  actually  used  In  the 
operating  environment 

AREA  OF  RISK 

A  distorted  view  of  the  system's  suitability  may  be  piesented. 

Inappropriate  or  inadequate  parameters  may  be  selected.  The  appropriate  number  of  parameters 
should  selected  to  provide  decisionmakers  with  a  complete  picture  of  die  system's  suitability. 
In  addidon,  the  test  should  allow  the  coltecdon  of  adequate  data  to  establish  confidence  in  the 
test  results.  In  the  early  stages  of  development,  the  opoadonal  suitability  elements  may  not  be 
fully  developed  and  the  maturity  level  may  be  inadequate  for  operational  testing. 

OUTLINE  FOR  REDUCING  RISK 

Selected  suitability  parameters  should  be  related  to  the  system’s  operational  requirements.  This 
allows  the  test  team  to  define  testing  that  provides  measures  for  the  evaluation  of  c^iatiraial 
suitability  issues. 

a.  Suitability  Parameters. 

Are  the  suitability  parameters  representative  of  those  used  in  the  actual  operating 
environment? 

'j  h.  suitability  parameters  in  the  OT&E  plan  should  be  consistent  with  the  operational  suitability 
i- «  -s  as  developed  in  the  TEMP.  Multiple  parameters  may  be  required  to  adequately  evaluate 
some  of  the  suitability  elements.  Additional  measures  may  be  necessary  for  each  support  level 
Volume  I  of  the  Operational  Suitability  Guide  discusses  suggested  suitability  parameters. 


To  measure  the  maintainability  oi  support  equipment  and  its  capability  to 
support  the  mission  in  the  tactical  environment,  the  parameters  will  be  mean 
corrective  maintenance  time  (MCMT)  at  each  level  of  maintenance,  maintenance 
man-hours  required  per  hour  of  operation  (MMH/OH),  and  distribution  of  the 
maintenanr^  workload  and  required  maintenance  capability  at  each  echelon  of 
maintenance  support.  The  need  for  special  tools,  test  equipment,  and  skills  will 
be  examined. 
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b.  Perfonnaiice  and  Supportability  Parameters. 

Is  there  a  sufficient  mix  of  performance  and  snpportabilit;  parameters  to  provide  the 
decisionmakers  with  information  about  the  to^  sjrstem,  as  well  as  the  support 
environment? 

The  test  should  exercise  the  system  in  its  intended  missions,  preferably  its  combat  missiems  as 
well  as  secondary  missions,  llte  maintenance  and  support  procedures  used  in  the  test  should  be 
representative  of  those  diat  will  be  used  when  the  system  is  fielded.  The  suitability  parameters 
must  be  fiilly  defined  in  the  test  plan  or  in  reference  documents  (e.g.,  the  TEMP).  Failure  defini¬ 
tions  and  scoring  criteria  also  must  be  included  in  die  ^iprtqniate  test  documents. 


One  operational  suitability  parameter  will  be  mean  time  between  operational 
mission  failures.  Data  collected  as  a  result  of  satisfying  this  requirement  also 
will  be  used  to  evaluate  the  mean  time  between  corrective  and  preventive 
maintenance.  In  addition,  maintenance  pubKcation  procedures  will  be  evaluated 
as  to  their  adequacy  to  support  preventive  or  corrective  maintenance. 


c.  Data  Efements. 

Have  required  suitability  data  elements  been  identified  that  will  allow  for  the  evaluation  of 
each  suitability  objective? 

The  test  plan  should  identify  the  data  elcruents  requiied  to  suppon  the  objectives,  measures  of 
effectiveness,  or  measures  of  petfonnancc. 


I 
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Test  constraints  or  limitations  can  prevent  or  degrade  the  assessment  of  opera- 
tiona  suitability  and  the  resolution  of  test  Issues.  Each  constraint  or  llrrritatlon 
will  result  In  objectives,  or  portions  of  objectives,  that  cannot  be  fully  assessed. 
Constraints  or  limitations  can  result  from  the  system  Itself,  the  availability  of 
resources,  environmental  conditions,  or  the  time  available  to  conduct  the 
operational  test 

AREA  OF  RISK 

Critical  opetational  suitability  issues  may  go  unresolved  during  OT&E. 

Constraints  or  limitations  may  not  be  identified  as  eaily  as  possible.  Alternatives  may  not  be 
addressed  in  test  planning.  As  a  result,  criticai  operational  suitability  issues  may  be  unresolvable 
during  OT&E  This  situation  can  result  in  systems  being  approved  for  fielding  with  significant 
potential  for  tqjerational  problems.  Limitations  must  be  l^wn  and  understood. 

OUTLINE  FOR  REDUCING  RISK 

It  is  not  feasible  firom  both  a  cost  and  test  realism  standpoint  to  design  a  vst  that  will  address 
every  conceivable  issue  that  should  be  addressed.  However,  once  constraints  and  limitations 
have  been  identified,  test  planning  can  be  focused  to  reduce  the  likelihood  of  significant  voids  in 
evaluation  inftmnaiion. 

a.  System  Configuratiaa. 

Is  the  equipment  to  be  tested  representative  of  the  ptanned  production  configuralioa? 
What  will  be  the  impact  on  operational  suitability  evaluation? 

The  test  plan  should  indicate  the  configuration  of  the  test  articles  and  compare  them  to  the 
planned  production  configuration.  The  system  to  be  tested  may  represert  only  a  portion  of  the 
qrs^  that  is  plarmed  for  eventual  operation.  If  significant  differences  exist,  then  the  potential 
Imtations  to  the  suitably  testing  should  be  highli^.ted.  Differences  in  configmation  may  lead 
to  differences  in  reliability  levels,  or  an  inability  to  evaluate  other  suitability  elements  such  as 
support  equipment  or  documentation. 


According  to  the  system  test  plan,  the  space  segment  is  not  ye!  fully  operational. 
Only  five  or  six  satellites  may  be  available  during  short  periods  of  the  6-hour 
window.  A  minimum  of  four  satelHfes  is  required  fo  obtain  precise  three- 
dimensional  position  fixes  as  required  by  some  users.  The  assessments  of  sysfem 
availability  issues  will  be  limited  due  to  the  limited  total  operating  time. 
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b.  Test  Resources. 


Are  the  necessary  suitability  test  resources  planned  and  scheduled? 

The  test  plan  and  other  program  or  test  documentation  should  he  reviewed  to  understand  the 
suitability  resources  ^uired  to  conduct  the  test.  The  personnel  selected  to  ctmduct  the  test 
should  be  representative  of  the  operators  and  maintaineis  of  the  system  mice  it  is  fielded.  The 
required  level  of  training  for  operators  and  maintainers  involved  in  the  testing  should  be  dis¬ 
cussed.  The  personnel  sdected  to  operate  the  system  should  be  representative  (e.g.,  experience, 
education,  and  career  fields)  of  those  who  will  operate  the  system  in  the  field. 


The  test  will  be  conducted  using  10  receiver  sets  of  the  Block  lA  configuration. 
The  personnel  involved  in  the  test  will  have  received  contractor-conducted 
'pilot'  training  courses  that  are  planned  for  the  operational  users  and 
maintainers  at  the  first  two  levels  of  maintenance. 


c.  Test  Scenaria 

Have  the  appropriate  geographical  settings  been  selected  for  conducting  the  test?  Where 
possible,  have  locations  been  selected  that  take  advantage  of  natural  settings,  as  well  as 
weather  conditions  (e.g.,  high  altitude  areas  with  snow,  as  well  as  hot  and  dry  locations)? 

The  selection  of  missions  and  test  sites  should  be  reviewed  to  ensure  that  cqiabilities  to  be  tested 
can  be  supported,  and  environmental  characteristics  such  as  climate,  terrain,  and  foliage  are 
lepresentative  of  the  planned  op^tional  environmenL  This  review  should  take  into  considcr- 
aiion  funding  constraints,  site  availability,  and  unique  site  capabilities. 


ff  the  system  is  planned  to  be  operated  in  geographic  settings  such  as  rieseil, 
tropical,  populated,  jungle,  and  arctic,  then  testing  should  be  condurXed  in  areas 
that  are  representative  of  these  geographical  settings. 


The  Test  Conduct  section  ol  the  OTtB  plan  should  describe  the  general  approach 
to  be  used  to  conduct  the  test.  Including  testing  for  suitability.  The  description 
should  Include  the  scenario,  envlrorunent,  threat,  tacBcs  and  doctrine  to  be  used, 
and  requirements  to  be  met.  It  also  should  highlight  constraints  or  IlmItaUons 
that  could  affect  the  realism  of  the  test. 

AREA  OF  RISK 

Data  that  are  essential  for  suitability  evaluation  may  not  be  provided. 

The  planned  test  may  not  be  lealistic  enough  to  provide  the  data  that  are  essential  for  suitability 
evaluation.  The  daita  must  be  useful  in  evaluating  the  system's  potential  suitability  for  the 
intended  operational  environmenL  The  OT&E  test  pian  also  may  be  of  less  detail  than  is  needed 
to  provide  an  understanding  of  the  specific  actions  i^uiied  during  the  conduct  of  the  test 

OUTLINE  FOR  REDUCING  RISK 

The  discussion  of  the  suitability-related  testing  should  be  broad  enough  for  the  reader  to 
understand  the  realism  of  the  test  Test  scenarios  should  be  planned  considering  the  opnadons 
that  will  be  required  of  the  system  in  combat,  as  well  as  the  types  of  threat  it  is  Ukely  to 
encounter.  Personnel  operating  and  maintaining  die  s^^m  during  testing  should  be 
rei^sentative  of  those  who  will  operate  the  system  once  it  is  deployed.  Personnel  should  be 
trained  in  advance  of  the  test  and  be  thoroughly  familiar  with  the  test  plan. 

a.  Test  Scenaria  (see  35.1} 

Are  the  factors  and  conditions  of  the  test  scenario  representative  of  those  that  will  be 
present  in  the  actual  operating  environment? 

The  plan  for  operational  suitability  testing  most  take  into  consideration  the  range  of 
environments  (tactical,  climatic,  etc.)  that  the  system  will  be  eiqxrsed  to  when  operationally 
eiiqtloyed.  Fiom  an  operancmal  suitability  stand^int,  it  is  impoitant  to  demonstrate  the  ability 
to  support  the  system  using  the  planned  support  structure  in  various  opcnitmg  conditions.  If  un¬ 
realistic  test  missions  are  lued,  then  the  suitability  assessment  can  be  t^timistic  or  unfavorable  to 
the  system.  The  missions  should  be  reviewed  to  ensure  that  the  factcns  and  conditions 
sufficiently  portray  the  intended  operating  environioent  and  Ktapo. 


The  land  navigation  user  equipment  tests  will  be  conducted  along  with  some  HELO 
tests.  Missions  will  be  conducted  with  representative  operational  user 
personnel.  The  aviation  set  will  be  tested  in  the  UH-60A  configuration  Test 
events  will  be  conducted  in  accordance  with  the  operational  mode  summary  and 
mission  profile  of  representative  operational  users  The  support  for  the  first 
two  levels  of  maintenance  will  be  in  accordance  with  the  planned  support  concept. 


Page  3-16 


OT&E  PLAN  -  Chapter  3 


b.  Test  Articles,  (see  3.1) 


Does  the  OT&E  plan  Indude  an  adequate  description  of  the  number  and  configuration  of 
the  test  artides? 

On  major  weapons,  the  test  article  normally  omsists  of  “worked  over”  prototy]^  therefore,  test 
articles  are  a  (tiffetent  configuration  and  some  are  without  subsystems.  Test  article  configuration 
should  be  examined  to  determine  its  effect  on  the  test  scenario,  realism,  and  data  collection.  Test 
articles  often  are  early  prototypes  and  often  will  not  be  representative  of  the  fielded  system. 
Thne  are  exanqtles  whm  hundreds  of  modifications  have  been  made  to  test  article  configura- 
titm  befme  the  systems  are  fielded.  The  Staff  Assistant  may  wish  to  estimate  tire  impact  of  these 
configurations  mi  the  system’s  evaluation. 


Prototypes  of  a  new  aircraft  have  been  developed  by  the  two  competing 
contractors.  Developmental  testing  has  been  progressive  and  both  contractors 
consider  their  prototype  aircraft  to  be  ‘representative*  of  the  system  to  be 
produced.  Several  subsystems  (including  the  weapon  release  system  and  the  on¬ 
board  diagnostics  system)  are  being  developed  and  are  expected  to  be  In  an 
operational  configuration  in  the  eighth  production  aircraft.  An  aircraft  with  all 
systems  completely  representative  of  the  final  configuration  will  be  available 
when  the  tenth  aircraft  is  delivered  in  another  three  years.  A  DAB  to  approve  the 
beyond-limited-rate  initial  production  is  planned  in  18  months.  (A  total  of  50 
production  aircraft  are  planned.)  Consequently,  the  OT  community  is  faced  with 
the  dilemma  of  a  ‘concurrent*  program  structure  wherein  a  complete 
operationally  representative  system  will  not  be  available  for  actual  field  tests 
before  the  scheduled  Milestone  IIIB  decision. 


c  Test  Hours,  (see  3.5  J!) 

Are  the  planned  test  hours  sufficient  for  an  adequate  operational  suttaltility  evaluation? 

The  number  of  test  hours  required  to  produce  the  necessary  daa  to  evaluate  ppetanonal  suitabili¬ 
ty  is  dhectiy  dependent  on  the  system’s  characteristics  a^  the  rrtissioa  to  be  p^ormed.  For 
higji  reliability  systems,  the  nurabm  of  test  hours  required  to  verify  die  level  of  reliability  may  be 
quite  hi^  In  addition,  if  Cew  maintenance  actions  are  to  be  performed  during  the  OT  activities, 
it  win  be  difficult  to  provide  a  complete  evaluation  of  afi  suitability  elements.  The  OTA  should 
justify  the  adequacy  of  the  lest  hours  available  for  the  OT. 


The  number  of  test  hours  scheduled  for  the  system  OT  was  Imited  due  to 
restrictions  on  test  funds  and  limited  spare  parts.  Had  the  OT  been  conducted  as 
planned,  the  spares  and  test  hours  may  have  been  sufficient;  however,  because 
there  was  a  test  schedule  slip,  the  hours  were  not  sufficient  to  support  the 
prolonged  test  schedule.  The  Test  Director  made  a  determined  effort  to  execute 
the  test  as  planned.  He  was  wiling  to  except  test  data,  although  some  were  of 
Kmited  value.  Hours  became  the  main  driving  torce  in  the  test,  compelfing  the 
Test  Director  to  except  a  lot  of  unusable  data.  The  SA  should  ensure  that  test 
hours,  as  with  any  test  resource,  are  not  the  driving  force  in  test  execution.  He 
should  ensure  that  data  collection,  test  reaRsm,  and,  only  then,  schedules  are  the 
primary  considerations  in  the  OTSE. 
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ST  SCENAmO 


A  test  scenario  Is  a  set  of  circumstances  by  erhlch  a  system  is  tested  In  a  repre¬ 
sentation  of  its  Intended  operational  and  support  environment  The  test  scerarlo 
should  be  based  on  mission  scenarios,  support  concept,  critical  operational 
Issu^,  objective,  and  test  limitations.  Consideration  must  be  given  to  terrain, 
weather,  and  other  operational  factors. 

AREA  OF  RISK 

The  test  scenario  nuy  fail  to  reveal  operational  suitability  deficiendes. 

The  operational  test  scenario  may  lack  the  detail  and  realism  needed  to  reflect  the  intended 
operational  use  of  the  system.  In  addition,  it  may  not  provide  for  adequate  (iteration  of  die  total 
s^tem  to  exercise  realisdc  demands  on  the  supptHt  system. 

OUTLINE  FOR  REDUCING  RISK 

.Scenarios  should  be  based  on  realistic  factors  and  conditions  that  will  be  present  in  the  intended 
operating  environmenL  Mission  roles  and  types  should  be  u^d  and  varied  in  such  a  maimer  to 
ensure  that  test  objectives  and,  ultimately,  the  critical  suitability  issues  can  be  answered  with  an 
acceptable  degree  of  confidence. 

a.  Test  Scenaria 

Is  the  test  scenario  representative  of  the  planned  operating  scenario  and  brood  enough  to 
allow  for  sufficient  data  collection  to  analyze  saitalHlity  issues? 

The  degree  of  realism  of  the  test  scenario  and  mission  determines  the  extent  to  which  the  sujqxxt 
system  and  qieTational  suitability  can  be  evaluated.  If  the  missions  used  arc  cnicalistic^y 
severe,  then  suitability  may  appear  to  be  less  than  it  really  is.  If  the  missions  are  less  severe  than 
the  expected  operational  missions,  then  the  evaluation  of  operational  suitability  elements  may  be 
optim^c  congiared  to  what  is  acUevable  in  the  operational  environmenL 


The  system  tests  will  be  conducted  with  a  cadre  ot  command  and  control 
personnel,  equipment,  and  sufficient  user  units  and  operators  to  bring  the 
community  up  to  370  user  units.  There  will  be  no  opposing  force  except  for  the 
EW  jammers  during  this  period.  The  scenario  calls  for  two  days  of  scripted 
defense  arxl  two  days  of  scripted  attack  over  approximately  the  same  area  as  the 
Phase  I  test.  Tests  will  be  conducted  12-hours  a  day,  which  will  allow  data 
processing  overnight  prior  to  continuing  the  next  day  of  testing. 


b.  Test  Events. 

Do  the  test  evoits  represent  the  minimum  number  of  mission  types  that  can  be  expected  to 
be  performed  with  the  system?  Are  affected  operating  activities  identified  and  test  events 
pfauuied  for  each? 
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The  planned  testing  should  include  a  leptesentative  set  of  the  planned  missions.  There  should  be 
planW  test  events  for  all  system  functional  elements;  they  should  involve  all  mission  types. 
The  events  must  be  representative  to  ensure  that  suitability  can  be  evaluated. 


As  a  minimum,  16  corridor  and  six  linkup  aviation  mission  types,  under  nap-of- 
the-earth  benign  conditions,  will  be  accomplished  using  nine  pre-surveyed  way- 
points.  In  addition,  a  minimum  of  six  corridor  and  six  linkup  mission  types, 
under  nap-of-the-earth  electromagnetic  conditions  using  nine  pre-surveyed 
waypoints,  will  be  accomplished.  All  operational  missions  will  include  en-route 
navigation,  present  positioning  data,  and  termination  fixes. 


c.  Maintenance  Concept 

Is  the  maintenance  and  support  concept  to  be  used  during  the  testing  an  accurate 
representation  of  the  operational  environment? 

Maintenance  support  to  be  provided  during  testing  should  be  representative  of  bow  maintenaiice 
be  performed  after  t^  system  is  fielded.  Operatkmal  suitability  elements  (e.g.,  test 
equipment,  maintenance  documentation,  and  man-machine  interfaces)  that  impact  the 
effectiveness  and  efficiency  of  maintenance  operations  should  be  included  in  the  evaluation. 


The  system  has  been  designed  to  use  the  estabished  five  echelons  of  maintenance 
support.  During  testing,  first  echelon  maintenance  will  be  performed  by  the 
radio  operator  for  all  ground  configurations  and  by  the  organizational 
rnaimenance  activity  of  each  user  of  the  airborne  radio  set.  Second  echelon 
maintenance  will  be  performed  by  the  Electronics  Maintenance  Company  and  will 
consist  of  operating  and  performing  checks  to  determine  fault  areas  prior  to 
evacuating  to  third  echelon  maintenance.  Third  echelon  maintenance  shall  be 
perfonned  by  the  contractor  and  will  ix>t  be  evaluated  in  this  phase. 


d.  Test  EnvironmenL 

Were  all  potential  operating  environments  considered  in  the  construction  of  the  test 
environments? 

The  planned  opeia^g  environment  can  be  described  in  many  ways,  including  weather  and 
geognqthical  cmiditions,  electromagnetic  conditions,  and  battle&ld  conditions  including  smc&e, 
noise,  and  CBR.  The  operanonal  testing  usutdly  is  limited  in  the  lan^  of  environments  that  can 
be  addressed.  Thaefore,  it  is  inqxntant  to  assure  that  those  conditions  that  potentially  could 
have  adverse  impacts  on  the  operational  suitability  of  the  system  are  addressed.  Adequate 
consideration  of  these  factors  sh^d  be  included  in  test  scenarios. 


The  Air  Force  testing  of  the  GPS  UE  requires  support  from  the  Air  Force 
Electronic  Warfare  Center  to  test  the  susceptibility  of  the  5-channel  airborne 
set  to  jamming  of  the  satellite  rfownfink  signal.  The  Air  Force  will  provide  the 
technical  expertise  on  the  selection  and  operation  of  an  airborne  jammer.  The 
jammer  will  be  operated  in  accordance  with  Warsaw  Pact  radio  electronic  combat 
doctrine  during  six  bombing  missions,  and  three  level  and  three  loft  profiles. 
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The  number  of  test  hours  planned  for  the  system  under  test  Is  directly  Influenced 
by  the  mission  types  and  roles  that  the  system  must  perform.  The  length  of  time 
required  to  complete  a  mission  plus  the  number  of  missions  In  the  scenario 
should  be  representative  of  the  ntisslon  lengths  and  duration  that  will  be  required 
In  the  Intended  operating  environment. 

AREA  OF  RISK 

Operating  time  will  not  provide  representative  data  and  thus  will  lead  to  faulty  conclusions. 

The  number  of  test  houn  or  the  mission  duration  may  not  be  adequate  to  provide  the  necessary 
data  for  evaluation.  The  amount  of  test  data  that  is  available  for  evaluation  is  directly  tied  to  the 
number  of  test  hours.  To  provide  the  decisionmaking  authority  with  a  realistic  assessment  of  the 
operational  suitability  of  the  system,  careful  considmtion  must  be  given  to  mission  length  and 
duration. 

OUTLINE  FOR  REDUCING  RISK 

The  number  of  test  hours  should  be  suffident  to  demonstrate  the  system's  r^ierational  and 
physical  characteristics  that  can  impact  the  c^ierational  suitability  of  the  system.  The  test  hours 
should  take  into  consideration  the  impact  that  a  shon  test  will  have  on  statistical  confidence,  as 
well  as  the  long  term  effects  on  operational  suitability. 

a.  Hours  to  Demoostrate  Characteristics. 

Is  the  number  of  test  hours  suffidait  to  demonstrate  the  operational  suitability 
characteristics? 


Operational  suitability  characteiistics,  such  as  reliability  and  maintainability,  require  relatively 
lengthy  test  perirxls  to  adequately  demonstrate  their  atdiieved  levels.  In  addition,  otha 
operatiotuil  suitabili^  items,  such  as  technical  data,  have  many  alternatives,  sections,  or  paths 
that  cannot  be  exercised  in  a  limited  perirxl  of  rqrerational  testing.  The  test  plan  must  deal  with 
die  need  for  lengthy  test  periods  and  t^istic  levels  of  test  resources. 


A  limited  number  of  test  hours  wiil  be  available  due  to  small  numbers  of  host 
vehicles,  limited  satelfite  coverage,  and  a  compressed  test  schedule  To 
supplement  the  operational  testing  data,  additional  data  will  be  available  from  a 
number  of  platforms  not  directly  involved  with  this  dedicated  OT&E  This 
additional  operating  and  failure  data  will  be  used  to  expand  the  system  suitability 
database.  Careful  scrutiny  and  evaiuation  of  the  supplememai  test  data  wili 
increase  the  confidence  ievel  of  the  suitability  resuits. 
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b.  Long  Term  Effects. 

Is  the  amount  of  time  dedicated  to  testing  individual  test  articles  sufficient  to  evaluate  the 
longterm  effects  on  the  articles? 

Test  items  may  have  risk  areas  that  requite  knowledge  of  the  effects  of  lengthy  periods  of 
operation.  Although  stanstical  methods  sometimes  allow  the  testing  of  a  large  numto  of  articles 
for  relatively  short  periods  of  time,  there  is  still  a  possibility  that  unknown  risks  will  be  realized 
when  the  system  is  exposed  to  longer  periods  of  operation.  When  the  OT&E  plan  calls  for  short 
periods  of  testing,  the  effects  and  rislu  associated  with  use  of  the  system  for  longer  periods  of 
tune  must  be  considered. 


A  selected  number  of  the  lest  articles  from  the  OT&E  Phase  lA  will  be  provided  to 
support  the  test  The  use  condition  will  be  similar  to  that  employed  in  t'hase  lA. 
Fai  re  data  from  this  extended  period  of  operational  use  will  be  evaluated  to 
det '  nine  if  any  long-term  failure  modes  are  likely  to  exist.  The  results  of  this 
additional  operation  will  be  provided  as  an  appendix  to  the  OT&E  report. 


c.  Statistical  Confidoice. 

Is  the  test  time  sufficient  to  ensure  any  required  test  confidence? 

The  system's  r^ratioiuil  requirements  for  lelirdrility  and  maintainability  may  include  the  need 
for  confidence  levels.  Confidence  levels  also  may  be  requhed  as  part  of  the  repotting 
requirements.  The  length  of  the  operational  test  should  be  reviewed  to  assure  that  the  required 
level  of  confidence  can  be  achieved  The  plan  should  state  any  requirement  for  confidence  levels 
to  be  calculated  for  operational  suitability  measures.  DoD  323S.1-H  may  be  used  as  a  reference 
in  assessing  the  confi^nce  levels  and  test  times  included  in  the  OT&E  test  plan. 


The  mission  reliability  of  the  system  shall  be  measured  in  terms  of  the  mean 
time  between  operational  mission  failures  (MTBOMF).  The  point  estimate 
MTBOMF  that  results  from  the  scoring  conference  data  shall  be  used  to  compute 
an  80-percent  lower  one-sided  confidence  interval.  The  80-percent  confidence 
value  shall  exceed  the  requirement  stated  in  the  approved  ROC.  Operational 
mission  failure  will  be  determined  by  a  formal  scoring  conference  in  accordance 
wifh  the  approved  failure  definition  and  scoring  criteria. 
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This  section  describes  in  general  terms  how  the  testing  organization  plans  to 
collect,  organize,  reduce,  verify,  manage,  control,  analyze,  and  store  the  data 
needed  to  perform  the  evaluation.  Suitability  evaluation  requires  data  on  many 
aspects  of  the  system’s  operadon  and  on  the  support  elements.  The  data 
management  plan  must  provide  for  the  proper  collection  and  control  of  the  test 
data. 

AREA  OF  RISK 

Inappropriate  or  incorrect  data  may  be  used  for  critical  evaluations. 

Foot  data  management  can  result  in  ineffective  and  inefficient  data  collection,  control  and 
ledncdon,  and  data  evaluation  methods  that  are  unable  to  support  the  decisionmaking  process. 

OUTLINE  FOR  REDUCING  RISK 

The  suitability  data  gathered  during  testing  must  be  properly  managed  to  ensure  credibility  and 
validity  of  the  conclusions  about  the  system  being  tesu^  In  determining  the  proper  types  and 
amounts  of  suitability  data  to  be  collected,  and  the  way  in  which  the  data  are  controlled  and 
evaluated,  a  well  defined  data  management  plan  is  required.  This  plan  may  be  part  of,  or  a 
supplement  to,  the  test  plan. 

a.  Quality  Control. 

Does  the  data  management  plan  indude  quality  control  methods  to  ensure  incorrect  or 
inaccurate  suitability  data  are  elinunated? 

The  data  management  plan  should  make  provisions  for  tracking  the  data  from  initial  receipt  until 
final  input  into  the  master  database.  The  plan  should  include  procedures  for  collection  forms, 
data  ve^cadon,  data  reduction,  and  entry  and  editing  of  the  data. _ 

When  suitability  data  discrepancies  are  discovered  through  regular  quality 
control  checks  of  the  data,  the  data  will  be  returned  to  a  previous  step  in  the  data 
flow  process  for  resolution  Ultimate  control  of  the  data  is  the  responsibility  of 
the  quality  assurance  test  manager;  however,  the  day-to-day  quality  control  of 
the  RAM  data  is  the  responsitnlity  of  the  RAM  data  manager.  Data  collection 
personnel  will  perform  quality  control  checks  to  identity  missing  or  logically 
incorrect  entries.  Quality  control  during  the  data  reduction  process  will  be 
accompGshed  by  both  manual  and  automated  methods.  Data  managers  will  ensure 
data  reduction  procedures  are  consistent  across  missions  and  that  results  are 
checked  by  someone  other  than  the  person  doing  the  actual  data  reduction. 
Scmbbing,  correlation,  and  merging  of  automated  data  from  various  sources  with 
other  data,  both  manual  and  automated,  will  be  controlled  by  computer  programs, 
when  possible.  The  program(s)  will  be  verified  to  be  tunctionally  correct  prior 
to  the  pilot  test.  Data  entry  and  edit  routines  will  be  controlled  and  monitored  by 
computer  programs.  No  data  will  be  loaded  into  the  master  database  until  all  data 
entry  errors  identified  by  quality  control  procedures  have  been  corrected 
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b.  Data  Collection  and  Validation. 


Have  procedures  and  responsibilities  been  established  for  manual  and  automated 
suitability  data  collection?  Does  the  test  plan  or  data  management  plan  describe  the  forms 
that  will  be  used  in  the  data  collection  process? 

The  test  plan  or  data  management  plan  should  include  procedures  to  control  the  data  collection 
process  for  both  manual  and  automated  data.  These  procedures  should  address  how  the  data  ate 
to  be  organized,  reduced,  verified,  managed,  controlled,  and  stored.  The  process  to  include  or 
exclude  data  from  the  datkbare  should  be  described.  How  will  “no  tests”  be  identified?  The  data 
collection  forms  should  have  been  develqred  and  approved,  and  the  responsibilides  for  data 
collection  defined.  Test  persoimel  should  be  trained  in  manual  and  automatic  data  collection 
prior  to  the  start  of  the  test  Sections  4.2.c  and  4.7  also  discuss  data  collection  and  validation. 


A  data  collector  will  be  assigned  to  each  of  the  systems  under  test  for  the  purpose 
of  RAM  data  collection  and  to  track  equipment  operating  time  and  events  at  the 
time  of  a  RAM  incident.  RAM  data  forms  will  be  based  on  forms  from  the  RAMES 
management  information  system,  but  tailored  for  use  on  the  system.  The  RAM 
data  forms  also  will  be  used  to  collect  RAM  data  on  the  MX-379  test  set  and  other 
support  equipment  RAM  forms  will  be  supplemented  by  narrative  commentary 
on  the  acceptability  of  the  technical  documentation  and  special  tools  used  for  each 
maintenance  action. 


c  Data  Processing  and  Analysis. 

Are  data  processing  procedures  described  in  sufficient  detail  to  allow  assessment  of  the 
procedures? 

The  planiung  should  include  data  handling  procedures  for  both  manual  and  automated  suitability 
data.  The  procedures  should  include  the  necessary  quality  control  for  the  d^  flow  from  tiw  data 
collectors  until  it  is  finally  entered  into  the  master  test  or  suitability  database.  Procedures  for  an¬ 
alyzing  suitability  data  should  be  described  in  sufficient  detail  to  provide  an  undcrsonding  of 
how  the  data  will  be  reduced  and  used  in  critical  evaluation.  Equations  and  algmithms  should  be 
addressed. 


Data  management  personnel  will  log  all  manual  suitability  data  fornis  provided  by 
the  data  collector,  review  them  for  consistency  and  completeness,  and  add  any 
additional  information  that  may  be  needed  for  internal  control.  If  there  are 
problems,  the  forms  will  be  returned  to  the  data  collection  section  for  resolution. 
Once  the  forms  are  determined  to  be  correct,  the  data  will  be  entered  into  the 
microcomputer  database.  Automated  quality  control  and  edit  routines  will  ensure 
data  validity.  The  quicklook  performance  and  RAM  reports  will  be  generated  for 
analysis  and  validation  by  data  managers. 
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Chapter  4 

OT&E  OBSERVATION 


In  addition  to  reviewing  test  documentation  such  as  the  TEMPs,  OT&E  plans,  and  OT&E  re¬ 
ports,  the  DOT&E  St&  Assistants  have  an  important  responsibility  in  performing  on-site 
observation  of  the  actual  conduct  of  the  operation^  testing.  (Title  10,  U.S.  Code,  Section  138 
states  that  the  DOT&E  may  require  that  observers  who  he  ^signates  be  present  during  the  prep¬ 
aration  for  and  the  conduct  of  the  testing  part  of  a  DoD  OT&E.) 

Test  program  reviews  for  programs  assiped  to  an  individual  Staff  Assistant  should  be  planned 
to  ensure  that  appropriate  priority  is  assi^ed  for  the  coverage  required  for  specific  tests.  Con¬ 
siderations  include  which  test  events  require  a  DOT&E  presence,  and  the  degree  of  on-site  moni¬ 
toring  which  will  be  required.  Given  that  a  particular  event  is  to  be  covered,  the  level  of 
involvement  may  vary  considerably.  The  Staff  Assistant  may  arraitge  for  a  team  of  DOT&E 
observers  to  be  on-site  for  the  duration  of  the  test,  or  there  may  be  only  one  DOT&E  observer 
on-site  during  only  one  or  more  of  the  critical  testing  periods. 

Once  die  particular  test  event  is  identified  and  the  degree  of  DOT&E  coverage  has  been  decided, 
then  preparation  for  the  test  observation  is  required.  This  Chapter  provides  guidance  to  assist  in 
preparing  for  on-site  idsits,  and  it  provides  information  on  items  to  be  considered  while  at  the 
test  site.  Once  at  the  site,  there  are  general  areas  that  need  to  be  considered.  The  management  of 
the  test  site  should  be  examined,  and  there  are  specific  items  in  the  test  events  that  need  to  be 
compt^  to  the  discussions  in  the  test  plan,  ^ally,  documet  ^tion  that  is  us«l  during  the 
opei^onal  test  events,  the  personnel  that  are  involved  in  the  operation  and  the  maintenance  of 
the  system  under  test,  Ae  data  collection  program,  and  the  test  scenarios  also  must  be  considered. 

This  material  was  prepared  within  the  context  of  this  Operational  Suitabili^  Guide  and  has  as  its 
focus  the  oversight  of  areas  of  testing  that  impact  on  the  operational  suitability  results.  However, 
most  of  the  items  in  this  Chapter  apply  equally  well  to  the  testing  for  the  operational 
effectiveness  objectives. 

The  templates  that  follow  address  suitability  considerations  for  a  number  of  the  activities  that  are 
involved  in  observing  the  operational  testing.  These  suitability  considerations  are: 

4. 1  Planning  for  Test  Site  Visits 

4.2  General  Observations  at  the  Site 

4.3  Test  Site  Management 

4.4  Comparison  to  the  Test  Plan 

4.5  Documentation 

4.6  Test  Personnel 

4.7  Test  Data 

4.8  Test  Scenarios 
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77i/s  section  discusses  considerations  for  the  general  "test  observation"  aspects 
of  the  OTiE  oversight  responsibHihes.  Observe  the  actual  operational  test 
activities  Is  needed  to  provide  for  credible  anises  In  DOT&E-prepared  Beyond- 
LRIP,  annual,  and  other  DOT&E  reports.  It  also  provides  first-hand  knowledge  to 
support  other  DOT&E  responsibilities  such  as  review/concurrence  on  the  Test 
and  Evaluation  Section  of  the  Congressional  Data  Sheets  and  Selected 
Acquisition  Reports  (SARs). 

AREA  OF  RISK 

Poor  preparation  for  on-site  visits  may  result  in  invalid  conclusions. 

A  prime  contributor  to  this  risk  concerns  the  balance  between  the  amount  of  time  dedicated  to 
day-to-day  Pentagon  responsibilities  and  time  expended  in  the  field.  The  abUity  to  ensure 
accurate  and  cre^ble  DOT&E  reports  can  be  strengthened  by  first  hand  observation  of  test 
events.  For  the  on-site  visit  and  observation  of  the  test  activities  to  be  effective,  preparation  must 
be  made  prior  to  amval  at  the  test  site. 

OUTLINE  FOR  REDUCING  RISK 

a.  Avrareness  of  Current  OT  Schedule. 

Are  current  test  activity  schedules  available  in  the  DOT&E  office? 

Test  schedules  in  the  TEMP  as  well  as  detailed  schedules  for  discrete  test  events  are  subject  to 
long-  and  short-term  revisions.  Attendance  at  selected  Test  Plan  Working  Groups  (TPWG)  or 
Test  Integration  Working  Groups  (TPVG)  will  provide  valuable  insight  into  schedule  variabiUty, 
the  status  of  system  development,  and  the  readiness  for  operational  test  activity.  The  Program 
Manager,  the  OTA,  or  the  Service  Staffs  Program  Eienxtnt  Monitor  are  sources  of  updr^ 
activity  schedules.  Informal  dialogue  with  members  of  the  test  team  will  facilitate  an  ongoing 
cognizance  of  the  program  status,  at  tiroes  prior  to  notification  hrom  the  Service  staff.  Schi^ule 
awareness  should  include  significant  pretest  activity,  e.g..  Test  Readiness  Reviews,  in  addition  to 
actual  test  conduct.  DOT&E  staff  members  can  leverage  their  effectiveness  by  utilizing  all 
sources  of  information  including  the  operational  test  agencies,  program  offices,  and  using 
commands.  These  groups  assist  in  establishing  test  objectives,  iden^ying  needed  resources, 
providing  farums  for  discussion  and  resolution  of  test  and  imegraDon  subjects/problems,  and 
related  activities.  During  attendance  at  meetings,  there  is  the  opportunity  to  gain  better  insight 
into  underlying  issues  and  the  associated  potential  impacts  to  the  operational  suitability  of  the 
weapon  system  under  test. 


Test  schedules  were  changed  due  to  the  late  arrival  of  the  test  article,  time  re¬ 
quired  to  install  and  calibrate  instrumentation,  and  time  required  to  conduct  pre¬ 
tests  and  training  After  two  weeks  of  test  trials,  the  Data  Authentication  Group 
(DAG)  recommended  that  earty  Inals  be  rerun  due  to  the  poor  data  from  those 
events.  At  this  point,  the  test  was  4-1/2  weeks  behind  the  original  schedule 
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b.  Structured  Planning  for  Each  Test-Observation  Trip. 

Will  the  limited  on-site  time  be  used  effectively  by  the  test  monitors? 

When  planning  for  a  trip  to  observe  test  operations,  an  activity  checklist  is  extremely  usefiiL 
This  checklist  will  assist  in  test  observation  activities  and  ptovi^  reminders  of  critical  issues  or 
risks  identified  in  key  source  documents  as  well  as  those  surfaced  in  prior  OSD  reviews  and 
congressional  interest  items.  This  approach  ensures  a  consistency  of  purpose  and  maximines  the 
benefit  of  the  on-site  activity.  Held  observations  will  include  deviations  from  plaimed  activities 
and/or  discrepancies  that  could  inrpact  quality  of  test  restrlts. 


The  Staff  Assistant  prepared  a  checklist  of  pertinent  information  prior  to  the 
scheduled  visit  to  the  test  sites: 

Configurations 
Hardware 
Software 

Critical  Operational  Issues 
Test  Oesign/Methods 
Data  Management 
Environmental  Considerations 
Intelligence/Threat  Information 


Interagency  Coordination 
Lessons  Learned 
Analysis  Procedures 
Management  Plans 
'’olicy  Documents 
Security  Requirements 
Simulation  Certification 
Test  Operations 


&  Preparation  for  Accurate  On-«ite  Observation  Reporting. 

Will  the  DOT&E  observers  recall  and  accurately  report  what  actually  happened  during  the 
monitored  activity? 

An  accurate  account  of  the  monitored  activity  is  crticial.  Careful  notes  must  be  taken  to  assure 
that  incorrect  uifotmation  or  conflicting  information  is  not  rqponed.  In  some  cases,  it  may  be 
desirable  U)  use  a  portable  tape  leco^/dictation  machine  as  the  test  event  progresses;  this 
would  be  supplementary  to  voice-tapn  (which  may  be  recording  radio  transmissioas,  etc.)  used 
by  test  participants.  Particular  attention  should  be  given  to  recording  the  specifics  of  unusual 
events  turd  those  which  caused  deviation  bom  plarmed  scenarios.  Test  monitors  should  maintain 
a  log,  recording  the  conditions  which  existed  dining  the  events,  or  scenarios. 


The  OCTSE  Staff  Assistant  used  a  hand-held  cassette  recorder  to  record  her 
observations  during  the  testing.  These  tapes  were  reviewed  each  everting  for 
major  significant  points  to  be  included  in  the  trip  report.  Upon  return  to  the 
Pentagon,  portions  of  the  recordings  were  transcribed  completely  by  the  DOT&E 
administrative  support  staff.  This  method  of  recording  test  site  observations 
proved  to  be  more  thorough  than  notes  or  personal  memory. 
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The  benefits  of  personal  observations  at  the  operationai  test  site  are  many.  The 
Staff  Assistant  (SA)  may  discover  information  that  wouid  not  have  been  thought 
important  enough  to  be  inciuded  in  the  OT&E  report,  but  that  wiii  be  important  in 
the  deiiberations  on  the  meaning  of  the  OTiE  resuits.  The  SA  must  be  sensitive 
to  this  type  of  information  before  going  to  the  test  site,  and  have  a  sense  of 
priorities  about  what  observations  and  conclusions  are  important 

AREA  OF  RISK 

Unexpected  Hems  or  unusual  events  may  yield  test  results  that  differ  from  those  expected 
when  the  test  was  planned. 

Once  the  operaconal  testing  is  initiated,  unplanned  situations  will  occur  and  the  testing  that  was 
intended  by  the  people  who  reviewed  the  plan  may  not  be  accomplished.  This  situation  is  due  in 
part  to  the  fact  that  operational  testing  can  never  be  completely  described  in  the  written 
document  The  readers  of  the  test  plan  infonnation  all  will  bring  dinerent  interpretations  to  what 
the  written  word  means.  Once  the  test  begins,  it  is  common  for  these  differences  to  become 
known.  When  the  DOT&E  Staff  Assistant  visits  the  test  site,  i;  is  essential  to  identify  any 
significant  difference  between  what  was  "thought  to  be”  ami  "what  actually  is.”  This 
identification  is  not  for  the  purpose  of  redirecting  the  testing,  but  to  place  critic^  test  results 
within  the  proper  context 

OUTLINE  FOR  REDUCING  RISK 

a.  Monitors  Must  Watch  for  "Good  Intentions”  Deviations. 

Are  deviations  from  approved  test  documents  occurring  inadvertently  or  through 
overzealous  efforts  of  test  participants? 

A  part  of  die  DOT&E  responsibilities  is  to  designate  observers  for  the  planning  for,  and  conduct 
of,  the  testing  portion  of  OT&E.  Experience  in  pardcipadon  in  test  reviews,  readiness  briefings, 
post-test  debriefings  or  reviews  al.'o  can  be  key  elements  of  DOT&E  monitoring  activity.  Prior 
experience  can  reveal  that  deviations  have  been  highlighted  dunng  internal  DoD  reviews,  as  well 
as  by  agencies  such  as  the  General  Accounting  Wfice  fGAO).  DOT&E  representation  at  test- 
rela^  activities  can  identify  impending  or  actual  variances  so  that  appropriate  corrections  can 
be  completed  tnereby  ensuring  credibility  of  the  test  programs.  During  the  on-site  visit,  the  SA 
should  search  out  deviations  tom  the  test  plan  caused  by  instrumentation,  personnel  problems, 
or  schedules  changes  that  may  etoct  the  validity  of  the  test  results. 


During  an  operational  test,  the  Engagement  Line-ot-Site  System  became 
inoperative  and  estimates  lor  making  repairs  in  a  reasonable  time  were  not  good 
Due  to  this  failum,  the  Test  Director  elected  not  to  lun  additional  tnal  events  of 
the  system's  ability  to  detect  and  track  the  threat  system,  even  though  the 
system's  abifity  to  detect  and  engage  systems  was  a  COI  that  was  to  be  addressed 
during  the  test  The  Test  Director  st.Med  that  data  would  be  obtained  f.'om  models 
and  simulations  to  answer  these  issuc- 
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b.  Resolution  of  Uncertainty  and  Trip  Report  Preparation. 


Are  there  "observations”  that  the  Service  Test  IMrector  will  not  agree  with  from  an 
accuracy  viewpoint? 

Observations  may  be  reviewed  with  the  Service  Test  Director  to  assure  that  the  test  monitcu'  has  a 
complete  understanding  of  the  actual  circumstances  and  events.  However,  test  monitors  will  not 
allow  such  review  to  impact  tepmting  of  their  observations  m  their  consideration  in  subsequent 
DOT&E  analyses.  Upon  completion  of  the  monitoring  of  a  specific  test  event,  the  Staff 
Assistant  (SA)  should  comdirtate  with  others  who  monito^  other  aspects  of  the  same  testing, 
and  all  significant  obsovadons  should  be  consolidated.  Prior  to  deputing  the  test  location,  the 
SA  should  review  his  major  observations  with  the  Service  Test  Director,  and  any  resulting 
conflicting  conclusions  should  be  highli^ted  in  the  SA's  trip  reprxt.  A  dambase  of  significant 
findings  fiom  the  monitored  activities  should  be  compiled  by  the  SA  and  maintained  fm  use  in 
devclc^ment  of  B-LRIP  repotts,  atmual  reports,  etc. 


From  interviews  at  the  test  site,  the  SA  determined  that  the  mission  pilots  were 
tlying  practice  ‘safety’  sorties  over  the  planned  flight  profiles  on  the  day  prior 
to  the  OT  test  events.  These  practice  fights  would  have  skewed  the  test  data 
When  the  observation  was  discussed  with  the  Test  Director,  it  was  discovered  that 
while  the  'safety*  flights  were  being  made,  the  flights  did  not  involve  the  pilots 
assigned  to  the  OT  test  e/ents. 


c  Data  Authentication  and  Validation. 

Are  data  being  validated  and  authenticated  by  an  independent  group? 

When  required  by  the  test  conqilexity  or  data  volume,  a  Data  Authentication  Group  (DAG) 
should  be  formed  to  perform  data  evaluation  and  certification  of  engmeering  analysis.  Normally, 
the  DAG  is  independent  of  the  data  nianagement  and  quality  control  process  and  is  not  under  tte 
supervision  of  the  data  manager;  it  provides  a  levd  of  quality  zssurarKc  above  that  to  be 
expected  fiom  the  data  management  quality  ctmtrol  funetkm.  Each  DAG  should  be  tailored  to 
the  unique  requirements  of  die  test  There  should  be  a  procedure  that  includes  the  DAG’s  con¬ 
cept  of  operations,  mganization,  responsibilities,  validation  procedures,  repotts  to  be  produced, 
and  requhed  schedules. 


There  are  critical  windows  in  the  data  collection  effort.  Should  the  windows  be 
passed  without  appropriate  actions  being  taken  by  the  test  team,  valuable  data  are 
lost  forever.  During  the  test  ot  an  aircraft,  it  was  deteniined  that  much  of  the 
data  collected  from  the  MIL-STD-1553  data  bus  were  of  very  limited  value 
because  the  on-board  recorders  and  playback  systems  had  not  been  developed 
sufficiently  to  allow  access  to  the  event  data.  If  the  Test  Director/chief  of 
instrumentafion/chief  of  data  management  had  the  playback  machines  developed 
,  and  tested  in  sufficient  time  to  be  used  to  evaluate  the  data,  valuable  information 
I  could  have  been  gained  and  test  resources  preserved. 
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The  pn^ierman^ement  and  control  of  the  test  site  and  the  test  sUe  activities  are 
^sential  to  providing  the  discipline  necessary  for  an  effective  operational  test 
program. 

AREA  OF  RISK 

Poor  management  procedures  or  procedures  that  are  not  followed  can  lead  to  invalid  test 
results. 

Convening  the  OT&E  plan  into  specific  actions  lequhcs  test  site  management  and  operating 
procedures.  The  Staff  Assistant  should  assure  that  platming  has  been  p^ormed  for  adequate 
management  of  the  test  site  and  that  proceditres  have  been  issi^  that  formalize  the  platming. 

OUTLINE  FOR  REDUCING  RISK 

a.  Program  Documents. 

Have  the  pertinent  program  test  documents  been  analjraed? 

A  prerequisite  to  effixtive  monitoring  of  system  testing,  simulation  activity,  demonstrations,  or 
other  events  is  an  in-depth  understanding  of  etcisting  guidance  related  to  the  event  (derived  from 
the  TEMP,  Operational  Test  Plan,  and  mher  documents).  Retest  analyses  assist  in  clmifying  die 
imponant  test  parmieters,  develc^ing  linkages  to  tie  field  test  data  with  modeling/simulation 
data,  and  in  establishing  the  analytical  structure  to  perform  final  analyses.  An  early  item  would 
be  verifying  OSD  approval  of  the  TEMP  as  well  as  the  DOT&E  approval  of  the  operational 
testing  of  the  systera 


Prior  to  leaving  for  the  test  sita,  the  SA  made  copies  of  the  pages  from  the 
unclassified  test  plan  that  describe  the  test  evems  that  were  planned.  The  TEMP 
for  the  system  had  been  approved  six  rnomhs  earlier  and  agreed  with  the  OTSE 
test  plan  that  was  reviewed  three  months  earlier.  Major  suitability  deficiencies 
from  the  previous  test  phase  were  summarized  as  low  percentage  of  fault 
detection  by  the  diagnostics  subsystem,  the  immaturity  of  the  maintenance 
software,  and  very  high  mean  times  to  repair.  The  corrections  to  these  deficien¬ 
cies  were  to  be  verified  during  the  test  evems  to  be  observed. 


b.  Instmmenfaitioa  Plans. 

Have  instnimentalion  plans  been  analyzed  to  determine  their  effect  on  the  tactical 
scenario? 


A  prerequisite  to  effective  monitoting  of  a  system  lest  is  an  understanding  of  the  instrumentation 
duu  will  be  used  to  collect  the  data  required  to  evaluate  the  system.  If  not  carefully  planned  and 
managed,  instrumentation  will  tend  to  “dtive*  the  test  After  review  of  the  instnunentadon  plan, 
die  SA  should  be  satisfied  widi  die  pn^  balance  between  test  realism  and  the  requirements  fm 
instrumentadon. 
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During  a  helicopter  OT&E,  there  was  a  requirement  for  instrumentation  to  be 
calibrated  before  each  test  event.  The  calibration  was  accomplished  by  flying  the 
helicopter  close  to  the  instruments  within  minutes  of  the  start  of  each  trial.  This 
requirement  provided  a  tactical  advantage  to  the  tested  system’s  crew  because 
they  had  an  opportunity  to  reconnoiter  the  battiefield  and  it  reduced  the  tactical 
realism  of  the  test.  As  it  turned  out,  the  instrumentation  that  required  this 
canbrafion  was  not  required  for  the  test  and  the  events  were  evaluated  without 
the  use  of  these  data. 


c.  Data  Management  Plans. 

Have  data  management  plans  been  analyzed  to  determine  if  the  plans  conflict  with  test 
realism? 

Reliable  operational  testing  tequiies  test  realism.  Data  collection  may  tend  to  conflict  widi  test 
realism,  thmfore  there  must  be  a  compromise  which  maximizes  the  combmed  issue  of  realistic 
testing  and  thorough  data  collecboiL  The  SA  should  ensure  that  data  collection  and 
instrumentation  is  not  causing  the  lest  to  be  unrealistic.  The  goal  of  instrumented  data  collection 
should  be  clearly  specified  in  the  data  management  plan.  Although  a  realistic  percentage  of  the 
total  data  generated  must  be  develqred  to  ensure  pmper  analysis  can  take  plaix,  as  a  nmnimtim 
die  plan  should  identify  the  critical  items  for  coU^on,  to  include  a  minimum  quantity  of  each. 
In  some  cases  the  test  event  may  be  a  singular  event,  in  which  case  it  becomes  a  critic^  item  to 
collect  these  data  and  redundancy  should  be  planned  to  ensure  adequate  collection. 


During  the  operational  test  of  an  aircraft,  the  data  coilection  plan  stated  the  re¬ 
quirement  for  an  on-board  tape  recording  system  to  capture  MIL  STD  1553 
Databus  information.  The  recorders  required  the  crew  to  land  aircraft  every  25 
minutes  to  change  the  tape  in  the  recorder.  Tho  test  trial  scenarios  were 
developed  around  this  data  collection  requirement,  which  made  it  very  difficult  to 
execute  the  test  trials  and  collect  meaningful  data  from  other  data  points.  Test 
realism  also  was  limited  because  of  this  data  collection  requirement. 


d.  Data  Authentication  Group. 

Is  there  a  group  of  professhmals  established  to  validate  data? 

The  Test  Diiector  or  Prognun  Manager  often  will  establish  a  Data  Autiientication  Group  (DAG) 
to  ve^  and  validate  trot  data,  in  action  to  assisting  in  data  reduction,  quality  control,  and  the 
identifjwg  anomalies  in  the  system,  instrumentation,  and  test  data.  The  DAG  must  be 
indepe^nt  of  'lie  system  develqier  and  data  manager  and  should  report  directly  to  the  Test 
Director,  it  is  important  that  a  standard  tolerating  procedure  (SOP)  be  written  for  the  DAG  ^ 
that  this  SOP  be  closely  followed. 


During  the  second  week  of  the  vehicie  test,  the  Data  Authentication  Group 
announced  that  the  night  navigation  trials  data  that  already  had  been  collected 
were  sufficient  to  provide  a  'high  level  of  confidence’  evaluation  of  the  system's 
performance.  The  Test  Director  eiected  to  terminate  the  scheduled  additional 
night  navigation  trials  and  use  those  resources  in  another  area  where  confidence 
levels  in  the  data  were  not  as  high. 


Chapler4  -  OT&E  OBSERVATION 


Page  4-7 


The  actual  test  will  dlfler  from  the  planned  test  due  to  a  host  of  reasons.  As  with 
most  plans,  their  usefulness  ends  with  execution.  Howevmr,  test  planning 
documentation  has  an  added  value  to  the  test  community  In  that  It  serves  as  a 
check  sheet  for  the  operational  t^tar  during  execution  phase.  The  SA  should  de¬ 
termine  the  degree  of  variation  from  the  plans  that  die  tester  has  been  required  to 
take  In  order  to  conduct  the  test  These  unplanned  wladons  often  are  weak 
links  In  the  process  and  should  be  evaluated  carefully. 

AREA  OF  RISK 

Actual  test  is  dilTerent  from  planned  actindes. 

During  the  acmal  test  conduct,  there  may  be  room  for  on-the-scene  decisions  that  result  in  the 
test  b^g  significantly  different  from  that  envisicuied  when  the  test  plan  was  written  and 
approved.  TiKse  undefined  areas  need  to  be  examined  during  test  site  visits  and  an  understand¬ 
ing  gained  of  how  the  items  will  effect  the  resulting  test  data  sad  evaluation. 

OUTLINE  FOR  REDUCING  RISK 

a.  System  Coafiguration. 

1$  the  system  configuration  to  be  tested  the  same  as  that  identified  in  the  test  plan? 

In  some  situations,  the  system  configuration  planned  for  use  during  OT  either  is  not  available  or 
it  is  delivered  to  the  test  site  with  some  unexpected  deviatiem.  In  limited  cases,  there  is  a 
deviation  in  the  number  of  test  articles.  The  limited  assets  available  early  in  the  development 
program  may  result  in  a  different  number  being  available  at  the  test  site  from  the  numba  drat 
was  plarmed  The  Staff  Assistant  should  determine  what  the  system  configuration  deviations  are 
and  evaluate  their  irtrpact  on  test  results. 


The  system  configuration  of  the  three  prototype  helicopters  provided  for  the  OT 
were  different  from  one  another  and  significamiy  different  from  the  planned 
fielded  system.  Maintenance  personnel  training  was  difficult  due  to  these 
differences.  (More  than  16C0  modifications  were  made  to  the  prototypes  before 
a  fielded  system  was  produced.)  Significant  contractor  support  was  required  to 
maintain  the  systems  during  the  test.  Many  prototype  subsystems  were  of  such 
early  configuration  that  their  performance  was  difficult  to  measure. 


b.  Test  Limitations. 

Will  test  Bmitations  significantly  impact  the  test  results? 

The  SA  can  review  all  listed  test  limitations  to  ensure  that  they  do  not  impact  on  the  ability  of  the 
test  to  meet  the  stated  objectives.  The  test  limitaiions  should  be  clearly  explained  and  it  should 
be  determined  if  the  limitations  can  be  avoided  or  resolved  by  the  responsible  oiganization. 
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The  testing  of  a  combat  earth  mover  was  limited  by  the  types  of  soil  and  weather 
conditions  to  which  the  system  would  be  exposed  at  the  selected  OT  location.  After 
evaluating  the  system's  performance  at  the  test  site  (clay/sandy  soil),  it  was 
determined  that  it  would  not  be  possible  to  estimate  the  system's  earth-moving 
capabilities  in  rrxsky  and  other  types  of  soil.  Additional  OT  at  other  test  sites  was 
required  to  provide  the  necessary  evaluation  report. 


c.  Unusiial  Pretest  Procedures. 

Are  the  test  artides  or  support  equipment  subjected  to  uousual  pretest  maintenance? 

During  most  OT  periods,  there  are  test  events  spaced  throughout  the  testing  period  with  periods 
of  nriiiimal  activity  in  betweeiL  During  the  rm-site  observation,  die  range  of  activities  that  occur 
during  the  non-test  periods  should  be  assessed.  Are  the  test  systems  subjected  to  unusual  pretest 
maintenance?  Are  the  test  articles  to  be  included  in  the  testing  selected  fiom  a  "pool”  of 
available  assets?  If  so,  is  the  selection  representative  of  what  the  operational  commander  would 
do,  or  is  the  selection  likely  to  skew  the  resulting  test  data?  While  there  may  be  motivaticm  to 
maximiTe  the  use  of  scarce  test  range  time,  the  test  team  must  guard  against  the  use  of  unusual 
pretest  maintenance  that  skews  the  reliability  or  maintainability  data. 


During  the  operational  test  and  evaluation  of  a  mortar  system,  it  was  determined 
that  the  ammunition  casings  had  been  machined  rather  than  cast,  as  is  normally 
the  case.  By  machining  the  casings,  the  weapon  proved  to  be  very  accurate  al¬ 
though  the  ammunition  was  very  expensive.  The  mortar  is  an  area  weapon  and 
the  normal  method  of  manufacturing  ammunition  is  by  casting.  When  cast  am¬ 
munition  was  obtained  and  fired,  the  mortar  did  not  meet  accuracy  or  dispersion 
pattern  requirements  at  extended  ranges. 


d.  Maintenance  Activities  Reflect  Operational  Concepts. 

Is  troubleshooting  or  system  repair  activity  performed  by  representative  personnel? 

In  the  case  of  a  majw  defense  acquisition  program,  no  person  enqiloyed  by  the  contractor  for  the 
system  being  tested  may  be  involved  in  die  conduct  of  the  Gnitial)  OT&£  that  will  sat^  the  B- 
LRIP  reporting  requirements  of  Public  Law.  In  addition  to  pretest  maintenance  activities,  post- 
test  involvement  of  such  contractors  could  be  a  cause  for  concern  (an  example  being  if  contractor 
personnel  became  overzealous  in  "Scoring  (Conferences"  by  defending  the  cause/chaigeability  of 
a  system  anomaly). 


During  the  operational  pretest,  the  system's  contractor  monitored  the  missile 
system's  failures  and  made  corrections  on  spare  computer  circuit  boards.  Before 
the  start  of  the  OT,  the  contractor  inserted  the  modified  boards  into  the  test 
articles.  The  substitution  of  modified  circuit  boards  disabled  all  on-board  test 
instrumentation  and  caused  four  missile  shots  to  be  made  without  data  being 
collected.  The  uncoordinated  modifi(»ition  of  the  test  articies  during  pretest 
caused  extensive  delay  and  cost  to  the  test  program. 
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The  Ooeamentatlon  at  the  test  she  should  Include  the  operator  and  maintenance 
Instructions  and  various  supporhng  manuals,  as  well  as  manuals  relating  to  the 
system  software.  The  role  of  the  documentation  Is  to  provide  for  system 
operation  and  maintenmce  that  Is  consistent  and  that  can  serve  as  a  basis  for 
evaluation  of  the  system. 

AREA  OF  RISK 

Sjstem  documentation  is  not  evaluated,  nor  does  it  provide  the  foundation  for  realistic  sys¬ 
tem  operation  or  maintenance. 

When  die  operadonal  testers  are  not  using  the  intended  system  documentation  for  operation  or 
maintenance  procedmes,  then  there  is  little  assurance  that  die  data  firom  such  a  test  are  rqnesen- 
tative  of  wh^  will  occur  in  operational  use.  The  system  documentation  should  pr^de  a 
foundation  of  system  operation  that  renders  die  test  dara  usable  in  operational  evaluation.  On¬ 
site  observation  of  testing  should  include  an  eifoit  to  assure  that  the  documentation  is  being 
propeiiy  used  by  the  test  site  persooneL 

OUTLINE  FOR  REDUCING  RISK 

a.  Source  and  Status  of  Docnmentatioii. 

Is  system  operation  and  maintenance  documentation  in  a  mature  state  to  allow  use  during 
testing? 

System  operation  and  maintenance  documentation  noimally  is  develtqied  either  by  the  system’s 
developing  contractor  or  another  contractor.  If  development  is  not  begun  early,  die 
documentation  usually  is  in  draft  stage  at  the  beginning  of  CIT&E.  Often  docui^t  developeis 
will  attenqx  to  modify  the  documents  during  pretests  and  tester  training.  These  modifications 
usu^  re^t  in  signiftcant  changes  in  the  system’s  tolerations  or  maintenance.  The  SA  should 
review  the  system’s  oper^ons  and  maintenance  documents  and  detennine  dieir  state  of 
readiness  for  the  OT&E;  dtis  includes  the  assessment  of  the  effect  of  early  drafts  and/or  poorly 
premted  and  pamal  documents  on  die  ability  ct  the  crew  and  maintenance  personnel  to  p^oim 
during  the  test. 


During  the  training  phase  for  a  weapon  system  test,  it  was  determined  that 
operation  and  maintenance  documents  were  in  an  eaily  state  of  development.  The 
Test  Director  requested  the  system’s  contractor  to  extend  the  crew  and 
maintenance  training  to  compensate  tor  this  lack  of  documentation;  the  draft 
documents  were  used  in  training  and  during  the  operational  test.  During  the  test, 
many  crew/maintenance  errors  could  not  be  classified  as  to  whether  they  were 
caused  by  improper  training,  poor  reference  documentation,  or  personnel 
errors. 
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b.  Use  of  Maintenance  and  Operating  Documents. 

Are  test  players  using  the  operational  and  maintenance  documents  to  assist  in  the 
performance  of  thar  duties? 

A  good  indication  of  the  level  of  completion  and  the  confidence  that  the  troops  have  in  the 
docunentadon  is  their  decision  to  use  the  documents  during  the  test  When  it  is  determined  that 
the  documents  are  not  being  useo,  it  may  be  desirable  to  a^  the  test  player’s  opinion  about  the 
documents.  When  the  documents  are  available,  the  SA  should  detennine  if  they  are  presented  as 
being  complete,  and  if  they  are  considered  accurate  by  the  test  personneL 
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The  observation  of  the  operational  test  gives  the  Staff  Assistant  a  sense  of  the 
skill  level  and  quallhcatlons  of  the  personnel.  Direct  contact  with  nsembers  of  the 
units  or  test  team  also  will  provide  Information  about  how  representative  of  typi¬ 
cal  user  troops  the  operations  and  maintenance  personnel  are. 

AREA  OF  RISK 

The  use  of  unrepresentative  personnd  may  resu!t  in  invalid  data. 

If  the  personnel  who  arc  operating  the  system  during  OT  are  more  highly  skilled  than  will  be  the 
plann^  operationai  troops^  the  system  will  perform  better  than  might  be  expected  in  the 
operadonal  units.  Likewise,  if  the  maintenance  personnel  are  unusu^y  hi^y  skilled,  die 
suitability  of  the  system  may  be  optimisticaily  reprated.  In  the  visit  to  the  test  site,  the  Staff 
Assistant  needs  to  ascertain  if  the  sloll  levels  involved  will  effect  the  lest  data. 

OUTLINE  FOR  REDUCING  RISK 

a.  Tester  Training. 

Are  test  personal  properly  trained? 

Training  plans  and  certification  plans  for  test  personnel  should  be  developed  and  published  early 
in  the  Full-Scale  Development  Phase.  Etiras  by  test  persramel  us'ially  are  expensive  and  often 
cloud  the  reason  for  test  failure.  The  SA  should  ident%  the  training  p^ormed  prior  to  the  start 
of  test  and  letermine  if  additional  training  was  given  to  test  personnel  as  a  result  of  the  pretest 
events. 


During  the  pretest  trials  for  the  OT  of  a  helicopter,  it  was  determined  that  the 
test  system  crews  had  received  additional  training  on  aircraft  subsystems, 
whereas  the  crews  for  the  baseline  system  had  not  received  this  additional 
training.  The  subsystems  involved  basically  were  the  same;  therefore  it  was 
decided  that  the  baseline  crews  shojkJ  receive  the  same  blocks  of  instruction  as 
did  the  test  system's  crews  When  comparing  a  new  system  with  a  baseline 
system,  training  becomes  very  important  and  must  be  carefully  monitored. 


b.  Use  of  CoDtractors. 

Are  the  system’s  developmg  contractors  participating  in  the  OT&E? 

Personnel  from  the  developing  contraaor  are  prohibited  from  participating  in  the  conduct  of 
operational  testing.  During  v^  early  test  phases,  the  contractor  may  have  a  role  in  some  of  the 
suitability  areas,  e.g.,  performing  second  or  third  level  maintenance,  or  assisting  in  maintenance 
that  is  specific  to  the  developmental  systems.  If  there  is  some  level  of  contraaor  involvement, 
the  Staff  Assistant  must  examine  the  way  in  which  this  involvement  might  be  influencing  the  OT 
data  results.  Are  the  Service  maintenance  perronnel  "consulfing”  with  the  contraara  during 
routine  mamtcnance? 
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During  the  OT  of  an  aircraft,  the  contractor  at<d  subcontractors  were  allowed  to 
move  into  a  motor  pool  complex  near  the  test  site.  It  was  very  difficult  to  keep 
the  contractors  away  from  the  test  systems.  On  one  occasion,  a  contractor  was 
observed  pushing  a  military  maintenance  person  out  of  the  way  so  that  he  could 
make  adjustments  on  the  aircraft  engine. 


c.  Use  of  “Gold  Crews.” 

Are  the  crews  involved  in  the  OT  typical  of  those  who  would  be  expected  to  operate  and 
maintain  the  system  once  fielded? 

The  tom  "Gold  Crews"  is  used  to  identify  a  situation  vdiere  the  personnel  who  are  operating  or 
maintaining  the  system  ate  of  a  higher  sl^  level  than  the  crews  «4io  are  expected  to  operate  or 
maintain  thr  system  once  it  is  field^  Is  the  sldU  level  being  used  of  sudi  a  degree  tiuu  it  skews 
the  OT  data  a^  dm  evaluation  results?  In  other  cases,  die  stall  levels  of  maintenance  personnel 
in  the  field  units  ate  ptitnarily  mid-leveL  hi  this  case,  the  maintenaiice  personnel  in  the  test 
should  not  (t^)  he  “hi^y”  skilled. 


An  aircraft  test  team  was  formed  with  maintainers  fresh  from  the  operating  units 
of  predecessor  systems.  While  at  the  test  sits,  the  SA  determined  that  while  these 
troops  were  representative  of  the  operating  units,  over  75  percent  of  them  were 
assigned  the  7-tevel  skill  code.  In  the  operating  units,  only  25-30  percent  of 
the  personnel  are  7-tevels.  The  evaluation  of  the  OT  maintenance  data  must 
consider  that  some  of  the  maintenance  actions  that  were  performed  by  7-level 
personnel  would  be  performed  by  S-levets  in  the  operating  units. 


d.  Stress  Level  of  PcrsooiKi  During  Test 

Will  the  test  crews  be  placed  under  the  operational  stress  that  wonid  be  expected  under 
combat  conditions? 

The  stress  levels  that  are  present  during  the  operational  use  of  combat  systems  camiot  be 
plicated  during  operational  testing.  Wl^  this  statement  is  true,  it  should  not  be  taken  as  a  ra¬ 
tionale  for  avoi^g  the  replication  of  reaUsm-enerriy  fire  can  be  simulated  and  unexpected 
events  can  be  entei^  into  "fiee-play"  portions  of  tests.  The  hmnan  foctors  evaluation  of  stress 
levels  in  combat  requires  that  considiaation  be  given  to  all  of  the  items  that  are  unknowns  during 
the  actual  combat  use  Of  the  system. 


The  troops  used  in  the  OT  of  a  shoulder-fired  missile  were  being  evaluated  tor 
human  factors,  portability  of  the  missile,  firing  effects,  ability  to  guide  the 
missile,  etc.  The  crews  selected  for  this  portion  of  the  evaluation  should  not  have 
more  experience  firing  the  predecessor  system  than  wiR  the  typical  user  troops. 
If  the  user  troops  only  have  experience  with  missile  simulators,  then  the  crews 
used  in  the  OT  should  only  have  an  experience  level  that  is  representative.  Stress 
can  be  introduced  into  the  OT  by  having  the  user  troops  operate  for  long  periods, 
similar  to  the  projected  operational  use,  and  at  the  projected  pace,  or  tempo. 
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The  obsmvaUon  of  the  operational  test  site  provides  the  Staff  Assistant  with  an 
understanding  of  the  test  organUatlon’s  approach  to  data  collection,  reduction, 
and  reportUig.  The  data  collection  plan  will  Include  the  methods  for  collecting, 
marldng,  handling,  storage,  and  disposition  of  test  data,  as  well  as  the  training 
reqidred  for  the  data  collectors. 

AREA  OF  RISK 

Improper  data  management  may  result  in  valuable  test  resources  being  lost  or  invalid 
condusions  being  reached. 

A  poorly  wiinsn  or  executed  data  coliectioD  plan  may  result  in  the  wrong  data  being  collected, 
the  data  not  being  clearly  matlced,  the  loss  of  data  due  to  improper  storage  and  distribution,  or 
the  loss  of  data  during  the  reduction  process. 

OUTLINE  FOR  REDUCING  RISK 

a.  Data  CollectHm  Concept 

b  the  data  coUectioa  concept  for  the  systems  to  be  tested  rational  and  executable? 

The  data  collection  concept  should  provide  pertinent  infatmadon,  including  a  list  of  key 
personnel,  resources,  and  procedures.  The  concept  should  indicate  the  degree  to  which 
instrumented  and  manual  data  coUecdon  techniques  will  be  used,  with  a  discussion  of  why  each 
techni^  is  used  The  concept  should  discuss  the  quesdmmaires  and  structured  interview  fmns 
that  wUl  be  used  to  support  the  qualitadve  measures.  There  should  also  be  a  discussion  of  how 
and  when  opinions,  interviews,  and  observadcms  of  the  players,  controller,  data  collector,  and 
test  directorate  personnel  will  be  gathered 


The  data  collection  concept  for  a  missile  did  not  specify  the  forms  or  formats  to  be 
used  for  the  collection  of  lest  player  comments.  During  the  field  assembly  of  the 
missile,  it  was  evident  that  troops  could  not  assemble  the  missile  with  the 
equipment  provided.  The  Test  Director  halted  the  test  and  had  the  troops 
interviewed  to  clearly  identify  the  problem.  After  the  interviews  were 
conducted,  it  was  determined  that  various  interviewers  had  placed  undue 
emphasis  on  specific  aspects  of  the  problem  and  had  ignored  other  areas. 


b.  Data  Collection  and  Processing  System. 

Is  there  a  collection  and  processing  system  in  place  and  do  the  test  personnel  understand 
what  they  must  do  to  use  the  system? 

The  collection  and  processing  system  should  describe  bow  the  data  are  to  be  organized,  reduced, 
verified,  managed,  ctmtrolle^  and  stored.  The  test  plan  for  the  tested  system  ^  list  each  data 
reqi^ment  arid  its  means  of  collection.  The  SA  should  consult  the  test  plan  to  determine  if  col- 
letrion  and  processing  systems  are  arranged  according  to  plans  and  if  there  are  data  requirements 
that  will  not  be  collected  because  of  the  process.  The  SA  also  may  determine  whethn  the  dara 
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aie  properly  cnganized  and  stored,  according  to  collection  source.  The  data  flow  diagram  (in  the 
data  collection  plan)  is  a  good  source  for  information. 


During  the  test  of  a  armored  vehicle,  the  test  team  did  not  correctly  execute  the 
data  collection  plan  This  resulted  in  a  disorganized  data  collection  activity  with 
piles  of  data  forms,  questionnaires,  and  other  documents  not  being  properly  re¬ 
corded.  filed,  and  stored.  By  the  third  week  ot  testing,  the  Chief  of  Data  Collection 
was  unable  to  determine,  from  the  documents,  if  trials  were  conducted  at  day  or 
night,  what  the  specific  weather  condtions  were,  or  the  degree  of  slope  that  the 
vehicle  was  operating  on  during  the  trial.  As  a  result  of  this  improper  data 
management,  an  additional  week  of  testing  was  required  to  fill  in  blanks  in  the 
data  matrix. 


c.  Quality  Control  of  Test  Data. 

Are  the  test  data  being  independently  validated  in  a  timely  manner? 

There  is  a  need  to  quickly  validate  the  test  data  to  detennine  if  additional  test  trials  are  required 
to  answer  t^ieratioiul  test  issues.  The  SA  should  determine  if  the  dm  collections  are  a^uate 
to  suppoR  the  OT&E  repott  He  should  ensure  that  thoe  ate  checks  and  procedures  in  pla%  at 
the  test  site  to  preclude  or  detea  and  conea  errors  made  in  data  collection,  dm  entry,  siA  dm 
reduction.  The  procedirte  should  also  outline  Ae  process  for  making  required  cottections  or 
changes  in  test  Aua  and  how  the  audit  aail  for  mose  correcuons  will  be  maintained. 


During  the  pretest,  it  was  determined  that  data  colleaed  procedures  were  not 
validated;  it  was  debated  whether  the  test  should  begin  before  corrections  were 
made.  The  Test  Direaor  was  unsure  If  critical  data  were  being  accurately 
colleaed  during  the  pretest.  However,  because  of  the  resources  involved,  it  was 
decided  to  start  the  test  and  complete  vafidation  during  the  earty  stages.  Critical 
data  were  lost  from  the  trials  performed  in  the  early  days  of  the  test  The  test 
was  stopped  after  one  week  to  fully  install  the  data  colleaion  procedures. 


d.  Data  Integrity. 

What  are  the  establistaed  procedures  for  ensuring  the  maintenance  of  data  integrity? 

The  data  collection  plan  will  discuss  the  proc^  through  which  each  set  of  collected  dm  is  to 
pass  before  teaching  the  storage  medium  which  suppoits  the  Test  Director  and  the  evaluator. 
Dm  flow  diagrams  identify  where  dm  are  combing  wiA  odier  dam,  and  where  they  are  pro¬ 
cessed,  scored,  leotganized,  validmd,  or  otherwise  manipulated.  The  dm  collection  plan 
should  describe  the  data  manipulation  process  at  each  stq>,  along  wiA  rules  and  procedures  for 
manipulati'm.  The  SA  may  wish  to  review  the  data  collection  plans  and  to  include  the  flow 
diagnuns  to  ensure  that  cai^  consideration  has  been  given  to  dam  integrity. 


Control  of  the  data  was  lost  during  the  transfer  from  the  test  site,  through  a  suD- 
port  contraaor,  to  the  test  agency.  It  was  determined  that  an  audit  trail  was  not 
established  during  the  planning  for  data  collection,  transfer,  and  storage.  This 
situation  allowed  dm  to  be  reorganized  and  manipulated  without  consideration  of 
the  effec  s  of  these  activities  on  their  evaluation.  As  a  result,  some  data  could  not 
be  used  because  their  authenticity  could  not  be  assured. 
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While  vistting  the  test  ^e,  the  Staff  Assistant  should  examine  and  record  the 
details  of  the  test  scenario  that  Is  being  used.  The  comparison  to  the  scenario 
that  was  described  In  the  test  plan  Is  an  Important  part  of  the  onsite  assessment 
The  test  scenarios  should  provide  realism,  as  well  as  an  opportunity  for  test  per¬ 
sonnel  to  collect  data  on  the  system's  effectiveness  amf  stHablllty.  Operational 
test  scenarios  normally  are  well  planned,  but  di»  to  location.  Instrumentation, 
and  data  collection  requirements,  they  may  not  be  well  executed. 

AREA  OF  RISK 

Operadonal  control  often  is  lost  and  trials  become  force-on>force  tactical  maneuvers. 

The  Staff  Assistant  may  determine  if  operationa!  control  of  the  tactical  forces  is  maintained  to 
ensure  that  test  objectives  are  being  met  Sttnaiios  that  are  designed  to  support  the  evaluaUB-’s 
requirements  tnust  be  closely  followed  to  ensure  that  data  coll''cti(m  is  accomplished.  Tester 
control,  in  a  force-on-force  operational  test  is  not  always  easy  b&.iuuc  uacdcal  units  tend  to 
perform  in  direct  proportion  to  their  level  of  training  arrd  modvalion.  During  initial  trials,  troops 
will  be  highly  motivated  and  tend  toward  being  urrconttuirabie  in  their  ze^  to  “win  the  war.” 
However,  as  the  test  wears  on,  test  personnel  wUl  become  bored.  Therefore,  test  scenarios  have 
to  be  carefully  monitmed  to  ensure  that  troops  react  with  enough  “gusto”  to  provide  realism. 

OUTLINE  FOR  REDUCING  RISK 

a.  Operational  Uncertainty. 

Does  the  test  plan  dictate  that  a  level  of  operational  imceitainty  be  maintained  during  the 
test? 

To  conduct  realistic  operational  tests  on  some  systems  requires  that  actions  by  the  threat  systems 
be  uncertain  to  the  player  petsouiel.  This  can  mean  uncertainty  of  time,  location,  type  of  action, 
directioD,  etc.  The  SA  ne^  to  assess  the  ‘‘reasonableness”  of  the  threat  scenarios. 


The  enemy  attack  forces  were  scheduled  to  initiate  a  new  test  event  each  day  after 
the  forces  reached  the  exercise  area.  As  a  result,  the  players  had  a  rough  idea  of 
when  the  attack  would  begin  After  three  days,  this  scheduling  became  obvious 
and  the  Test  Director  took  control  of  initiating  the  attack  forces. 


b.  Approved  Doctrine  and  Tactics  Used  During  tbe  Test 

Are  the  tactics  and  doctrine  used  by  tbe  test  system  and  tbe  threat  systems  approved  by  the 
responsible  agency? 

There  usually  is  considerable  dsagreetnent  about  how  the  enemy  will  fight  a  new  weapon 
system.  For  exanqile,  in  a  high-inrensity  conflict,  will  the  enemy  attack  helktmter  be  d^loyed 
nfqMjf-die'^aith,  low-level,  or  at  altitude?  The  doctrine  and  tactics  for  engagements  of  these 
helicop'  -IS  will  depend  on  an  assessment  of  the  most  probable  methods  of  d^loyment  The  SA 
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dwuld  detenmne  if  dK  Service  has  clearly  established  doctrine  and  tactics,  based  on  the 
approved  STAR,  for  the  dqtloymrat  of  the  tested  system.  He  should  determine  if  the  test 
scenarios  follow  the  tqiprov^doctiiiie  and  tactics. 


During  the  testing  of  an  attack  aircraft,  threat  crew  members  decided  to  use 
tactics  that  did  not  confonn  to  approved  doctrine.  These  tactics  caused  wide 
divergence  in  the  pianned  scenarios  and  several  days  of  testing  were  disrupted 
while  the  crews  were  properly  briefed.  Proper  tactics  and  doctrine  were 
included  in  the  test  events  thereafter. 


c.  Tactical  Operations  Center  Activities. 

Is  there  a  tactkal  operational  center  located  ic  the  field  which  is  operated  by  trained 
tactica!  operations  personnel? 

To  mainra:;!  cocticl  of  tactical  forces  during  a  force-on-foice  opei^ooal  test,  it  is  iiqiottant  to 
have  a  fully  manned  tactical  opeiations  center  located  in  die  field  widi  die  openting  forces.  This 
center  sho^  be  manned  by  fully  trained  operadons  personnel  ate  derailed  fiom  a  tactical 
uniL  The  forces  must  be  carefully  briefed  on  the  scenarios  and  the  objeedves  cf  the  operadonal 
test  and  evalnadoii.  The  SA  r^ould  ciu:?e  that  these  personnel  understand  diat  good  cmtrol  of 
foe  maneuvering  forces  iiuist  be  maintained  m  order  te  die  lest  personnel  to  cidlM  the  data  re¬ 
quired  to  evaluate  the  system  being  tested. 


Prior  to  an  OT  event  that  included  an  armor  engagement,  the  tanks  being  tested 
where  driven  for  a  number  of  hours  to  simulate  a  cross-country  movemem;  as  a 
result,  the  systems  In  the  tanks  were  in  a  confotion  similar  to  what  they  might 
reflect  in  actual  use.  In  this  situation,  to  go  directly  from  maintenance  faciBties 
to  ihe  engagement  areas  was  not  realstic. 


d.  Tcstiiig  Terrain  and  CfimatkConditioiB. 

Are  cnvironmenUI  CKtois  for  the  OT  realistic  for  Uw  system’s  nasstan  profile? 

Because  of  the  availability  of  operadonal  test  ates,  it  is  vesy  difficult  to  find  the  desired  tenain 
and  climatic  condidons  in  o^ex  for  OT  to  bo  conducted  under  condidons  that  are  fully 
representadve  of  the  system’s  intended  deployod  operadonal  missiou  pnffile.  Also,  tredeoffs 
will  have  to  be  made  because  of  safety  and  operadonal  constroints.  The  SA  should  dmemnne  if 
foe  site  selected  for  the  test  is  representadve  of  the  terrain  and  cliinadc  condidons  required  to 
provide  realisdc  testing. 

During  the  Phase  X  OT  of  an  Army  vehicle  system,  heavy  rains  inundated  a  num¬ 
ber  of  test  site  areas.  A  test  evem  was  added  to  demonstrate  the  vehicle  recovery 
capabilty  and  the  associated  training,  equipment,  and  procedures. 
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Chapters 
OT&E  REPORT 


The  OT&£  Tcpoit  can  be  viewed  as  the  most  imponant  document  in  die  OT&E  process.  While  it 
is  true  that  the  TEMP  provides  the  foundation  for  all  of  the  operational  test  and  evaluantn  and 
the  test  plan  provides  the  details  for  the  information  in  the  TEMP,  the  OT&E  report  is  the 
product  of  all  the  test  planning,  management,  and  actual  conduct  of  the  T&E.  The  OT&E  repott 
is  the  document  that  provides  the  T&E  results  to  the  decisionmakers.  It  further  serves  to 
complete  the  process  and  reabze  the  purpose  of  operational  test  and  evaluation,  and  to  provide 
information  upon  which  to  base  decisions.  Without  clear  and  complete  reporting  in  the  test 
report,  test  planning  and  test  conduct  may  have  little  impact  on  the  acquisition  program  decision 
pnxxss. 

There  are  a  number  of  different  OT&E  repqm;  quick  locdc  reports,  informal  reports,  briefing 
rept^,  and  formal  reptm  for  different  acquisition  phases.  Each  of  t^se  OT&E  reports  must  be 
considered  for  its  individual  purpose,  and  the  pha%  the  acquisition  program  is  in.  The  report 
should  show  the  audit  trail  tbiough  die  program  rmlestones  and  present  the  program's  progress. 
It  should  summarize  the  planned  testing,  wlut  happened  during  die  test,  and  the  test  residts. 

In  reviewing  the  OT&E  report,  the  DOT&E  Staff  Assistant  has  two  objectives.  First,  he  should 
ensure  that  the  report  accurately  reflects  the  test  and  evaluation  that  took  place,  and  that  it  con¬ 
tains  current  complete,  and  accurate  data.  Second,  he  should  review  die  test  and  evaluation 
results  on  the  syst^  including  the  issues  and  deficiencies,  and  recommend  a  DOT&E  position 
to  be  reported  to  the  appropriate  decisionmaking  forum. 

There  is  no  standard  Department  of  Defense  ftxmat  for  an  OT&E  lepmt;  each  Service  has 
developed  its  own  approach  to  documenting  the  required  information,  ^tliin  the  Army,  there 
arc  two  separate  reports  for  OT&E  activity.  The  test  repott  (TR)  is  the  primary  record  of  the 
operational  test  and  the  findings  and  facts  that  tesulted  fiom  the  testing.  The  indqiendent 
equation  reptxt  (I^)  documents  the  evaluation  of  the  test  data.  These  two  documents  together 
provide  the  composite  information  that  is  provided  in  the  OT&E  reports  of  the  other  Services. 

From  an  tqreratioiial  suitability  standpoint,  the  operational  test  and  evaluation  rqrort  is  the  con¬ 
cluding  document  in  the  process  of  determining  if  a  system  is  suitable  for  ^relational  use.  The 
report  should  highlight  the  background  of  the  test,  ie.,  dK  set  of  circumstances  and  decisions 
tiiat  determined  what  testing  was  conducted.  Most  operational  testing  has  limitations,  so  it  is  im¬ 
portant  that  the  limitations  that  affect  the  suitability  results  are  iden&ed  and  documented  in  die 
OT&E  iqrorL  The  fiamework  for  repeating  die  testing  results  is  the  evaluation  criteria  that  were 
developed  prior  to  the  actual  testing.  These  criteria  include  die  suitability  critical  issues  and  die 
associated  characteristics,  parameters,  and  thresholds.  The  test  that  was  conducted  should  be 
summarized  and  any  unexpected  changes  fiom  the  testing  that  was  planned  should  be  identified 
and  documented.  Finally,  the  report  should  discuss  the  test  data  arid  results,  as  well  as  provide 
an  evaluation  of  the  system  and  conclusions  about  the  system’s  t^ieratitnial  suit^ility. 
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The  templaies  that  follow  address  each  of  the  major  content  areas  indoded  within  OT&E  repocts. 
These  areas  are: 

S.l  Backgtonnd  Issues 
S2  Limitatioiis 

5.3  Evaluation  Qiteiia 

5.4  Summaiy  of  Test  Conduct 

5JS  Test  Results 

S.6  Conclusions 

Table  S-1  was  prepared  ftom  die  Services'  policies  on  OT&E.  It  shows  how  the  stiucture  of  this 
opetational  suitability  guidance  document  relates  to  the  major  sectkns  of  the  respective  Services' 
OT&E  reports. 
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The  backgmund  Issues  Include  those  major  events  and  Information  that  are 
needed  to  place  the  OTiE  results  Into  proper  perspective.  Items  In  this  category 
Include  faogram  direction  on  the  conduct  of  the  OTiE,  and  the  residts  and 
limftatfons  of  earlier  operational  testing  as  well  as  significant  operational  Issues. 

AREA  OF  RISK 

Unclear  statements  of  issues  may  result  in  an  incomplete  or  biased  view  of  what  the  test 
results  mean. 

If  the  background  is  not  summarized  adequately,  the  testing  and  T&E  results  may  not  be  placed 
in  the  premier  perspective.  The  T&E  activity  cm  a  program  is  a  continuum  of  activity  that 
includes  pmo^c  reports.  To  gain  maximum  benefit  fitun  these  reports,  the  context  of  die 
testing,  tte  meaning  of  the  evaluation,  and  full  consideration  of  the  results  is  essentiaL  If  the 
purpose  of  the  testing  and  the  hackground  are  not  described  adequately,  the  reader  may 
formulate  an  incomplete  at  biased  view  of  what  the  test  results  mean. 

OUTLINE  FOR  REDUCING  RISK 

a.  Previous  Test  Phases. 

Are  the  important  operatianal  suitability  aspects  of  the  previous  test  phases  summarized? 

The  previous  test  phases  provide  the  fundamental  starting  point  for  die  test  diat  is  being  repott^ 
on.  The  surrunary  of  these  earlier  test  periods  should  include  die  impeatant  areas  that  were  satis¬ 
factorily  tested  and  also  summaiize  dm  significant  results.  It  should  include  which  operational 
suitability  areas  were  tested  well  and  bad  good  results,  which  areas  were  not  included  in  the  test, 
and  which  areas  were  tested  but  did  not  meet  their  requirements. 


The  previous  system  tests  were  summarized  as  they  apply  to  the  major  RAM 
characteristics.  The  reliability  chart  showed  the  stated  quantitative  requirement 
(in  mean  time  between  operaticnat  mission  failure  -  MTBOMF)  for  each  of  fhe 
three  configurations.  The  test  results  for  each  of  the  completed  OT  phase  were 
Rsted.  For  reliability,  the  point  estimate  value  and  the  value  at  the  80  percent 
lower  confidence  limit  were  shown  in  the  chart. 
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b.  Previous  Management  Dedsioas. 

Did  previous  management  decisions  drive  the  operational  suitability  test  planning? 


Arc  the  decisions  that  directed  the  test  and  die  issues  involved  in  those  decisions  described  in 
enough  detail  to  permit  the  reader  to  understand  the  context  of  the  decision?  Management  ditec- 
dtm  that  resulted  fiom  previous  testing  may  highli^t  critical  or  risk  areas;  understanding  these 
concerns  can  assist  in  placing  the  operational  suitabdity  results  into  piiqier  perspective. 


Management  review  of  the  previous  testing  results  focused  attention  on  the 
progress  in  the  diagnostics  area.  The  maturity  level  of  the  built-in  test  during 
the  previous  test  phase  was  less  that  expected.  This  area  was  highlighted  by  the 
Acquisition  Decision  Memorandum  as  an  area  of  risk  that  should  received  addi¬ 
tional  attention  during  the  next  OT&E  phase. 


c.  Operatioual  Suitability  Characteristics. 

Are  suitability  characteristics  induded  in  the  system  summary? 

The  system  summary  within  the  test  reptm  should  be  a  relatively  short  discussion  of  what  was 
tested,  emphasiring  the  mission  oi  function  of  the  system.  Any  system  attributes,  !.e.,  new  tech¬ 
nology  or  cqiability,  high  risk  areas,  etc.,  that  are  the  reasons  for  critical  opmtional  issues 
(COIs)  should  definitely  be  included.  The  suppm  ctmc^  also  should  be  sunmurized.  since  diis 
is  needed  for  assessing  how  complete  the  optional  suitalnlity  portion  of  the  OT^  was. 


The  system  summary  includes  an  outline  of  the  maimenance  concept  planned  for 
the  system.  The  summary  outlines  how  the  system  will  be  supported  at  each  level 
of  maintenance  and  what  maintenance  will  be  peiformed  at  each  level. 


d.  Systan  Differences. 

How  was  the  tested  system  different  than  the  plaimed  operational  system?  What  are  the 
resulting  implications  for  smtaltility  testing? 

The  systems  under  test,  parbculatiy  in  die  early  stages  of  OT&E,  may  be  significandy  differem 
from  the  planned  operational  system.  These  earty  systems  genoally  will  not  be  from  the  produc¬ 
tion  line  that  is  planned  for  the  full  rate  production  and  tbmfore  will  not  have  die  full  benefit  of 
sable  production  processes.  The  system  differences  should  be  discussed  as  they  relate  to 
limitations  on  the  suirability  testing  and  the  need  for  any  additional  suitability  testing  in  a  latrr 
phase. 


The  test  report  indicated  that  the  operational  software  that  was  availaole  during 
j  the  lest  was  not  the  version  planned  for  the  production  systems.  Because  of  this 
I  difference,  there  were  some  diagnostic  and  mainte’iance  operational  capabilities 
[  that  were  not  verifiable  during  the  opar,viionai  suitability  testing. 


Mi  major  t^ts  have  limitations.  These  may  Include  limits  In  the  operating  envi¬ 
ronment,  the  length  of  the  test  period,  or  the  type  or  number  of  test  units, 
personnel,  or  supporting  devices.  The  list  of  limitations  highlights  the  potential 
risks  involved  In  assuming  that  test  results  are  totally  Indicative  of  what  might  be 
expected  In  the  operating  environment. 

AREA  OF  RISK 

Insufficient  identification  of  limitations  may  result  in  inaccurate  assessment  of  the  OT&E 
results. 

Limitations  that  occur  during  the  actual  testing  may  not  be  highlighted  in  the  test  tepon.  Such 
limitations  could  result  from  the  lack  of  realism  of  the  test  environment,  the  test  duratkm,  die 
number  of  test  articles,  <x  the  availability  of  test  hours.  The  test  articles  mi^t  not  be  of  the  l^st 
configuration  or  capability.  Without  visibility  of  the  limitations,  suitability  implications  may  be 
overlooked.  The  risk  in  this  area  is  diat  limitations  are  not  identified  sufficiendy  for  the 
decisionmaker  to  know  how  to  assess  the  OT&E  results. 

OUTLINE  FOR  REDUCING  RISK 

a.  Operational  SuitabOily  Test  Limitations. 

Was  there  a  summary  of  all  known  test  limitations  that  would  affect  suitability? 

Operational  suitability  limitations  may  include  many  areas.  Limited  number  of  test  hours  can  be 
a  limitation  on  evaluating  the  level  of  reliability  and  maintainability.  Others  may  be  the  result  of 
the  logistics  supptm  du^g  die  test  period  being  not  representative  of  the  platuied  t^ierational 
support.  Test  equipment  may  not  be  available.  All  tqietational  suitability  limitatinns  should  be 
sutnmarized  to  per^t  the  reader  to  place  the  test  results  into  perspective. 


There  was  an  inadequate  number  of  test  hours  on  selected  host  vehicles.  A 
complete  reliability  evaluation  of  the  system  is  dependent  on  sufficient  test  hours 
to  provide  statistical  confidence  in  the  results.  Lack  of  fest  hours  and  a 
compressed  test  period  will  reduce  the  statistical  confidence  in  the  OT&E  results. 
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b.  Additional  LimitatioDS  During  Testing. 


Were  tliere  any  limitations  that  developed  during  the  conduct  of  the  actual  testing?  Are 
they  presented  and  their  impact  on  operational  suhahility  discussed? 

Additional  limitations  beyond  those  in  the  test  plan  may  suiface  once  the  test  has  started. 
Specific  test  articles  may  not  be  available  as  plann^  Other  program  priorities  may  dictate  a  re¬ 
vision  to  the  activity  that  was  planned  in  the  OT&E  test  plan.  The  test  e^pment  planned  for  tire 
test  site  may  not  be  available.  Other  items,  such  as  technical  doenmentatkm,  may  be  delayed  and 
not  included  in  the  tqierational  suitability  evaluation. 


The  software  for  resting  the  radar  system  components  on  the  XX-537  test  station 
was  not  availabie  at  the  test  site  as  planned.  As  a  result,  the  second-level 
maintenance  for  the  radar  was  not  included  in  the  test. 


c  Environmental  Differences. 

Were  there  significant  differences  between  the  test  environment  and  the  expected 
operational  environment? 

Test  limitatioas  could  result  for  significant  environmental  di^rences.  These  differences  could 
be  major  ftetors  in  assessing  the  acceptability  cf  the  operatioaal  suitability  results.  The  test 
tqxxt  should  identify  the  major  conditions  that  were  diffeiem.  This  could  include  die  ratio  of 
support  assets  to  test  articles,  die  skill  levels  of  the  maintenance  personnel,  or  the  dqpdi  of 
training  that  the  maintenance  personnel  received. 


The  mairUenance  team  that  supported  the  system  during  the  testing  was  generally 
of  a  higher  skill  level  than  the  planned  maintenance  organization.  Analysis  of 
specific  task  times  was  performed  to  rietermine  the  effect  of  this  skill  difference 
and  the  resulting  impact  on  the  measured  mean  time  to  repair. 


d.  SmUbDityElaiKnisNothitlieTcst 

What  suhahility  dements  were  not  aUc  to  be  evatoated  rliirii^  the  test? 

In  most  OT&E  phases  it  is  common  for  some  support  elements  to  ntx  be  present  eluting  die 
testing.  An  exan^le  is  second-level  test  equqmieat  that  is  not  yet  devdoped  to  die  stage  where  it 
is  rea^  for  the  OT&E  envirDumem.  Odier  elements  may  have  herns  subsdtutri  diat  are 
sigmfir^dy  difierent,  such  as  fiKXoiy  test  equipmem  used  instead  of  the  planned  operatitpd  test 
equipment  The  effect  of  these  situations  must  be  considered  to  ideatify  the  areas  of  suitability 
that  are  not  yet  evaluated.  These  risk  areas  should  be  consideted  by  die  rlecisionmakers  and 
induded  in  future  phases  of  test  and  evaluation. 


For  the  test  period  being  repotted,  the  second  level  of  maintenance  was  performed 
by  the  hardware  comractor  at  his  facifity  and  therefore  was  not  part  of  the 
evaluation. 
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The  evaluation  criteria  should  be  derived  from  the  requirements  of  the  user 
organiaatlotjs.  There  also  may  be  qualitative  criteria  for  some  of  the  operaUonal 
suitability  elements. 

AREA  OF  RISK 

If  the  reported  results  are  not  related  to  the  user’s  stated  requirements,  then  the  test  and 
evaiuatfam  report  may  not  be  meaningful. 

The  proper  evaluation  of  suitability  test  or  analysis  results  is  dependent  oa  the  pre-established 
criteria.  These  criteria  must  be  related  to  the  us^s  stated  requirements.  The  test  and  evaluadcm 
repent  may  not  be  meaningful  if  the  results  are  not  related  to  dhe  user's  stated  requirements. 

OUTLINE  FOR  REDUCING  RISK 

a.  Criteria. 

Were  there  established  criteria  for  each  of  the  operatioiud  suitability  characteristics? 

To  be  nManingful,  each  operational  suitability  characteristic  should  have  evaluation  criteria 
drawn  fiom  the  us^s  stated  requirements  and  stated  in  the  OT&E  test  pUm.  Was  enough  denul 
provided  to  define  the  criteria?  For  example,  is  there  some  indication  of  how  Mures  are 
defined?  Was  a  reference  listed  that  included  the  failure  definition? 


The  reliability  criterion  is  stated  in  terms  of  mean  miles  between  unscheduled 
maintenance  actions  (MMBUMA).  The  system  shall  achieve  200  MMBUMA. 


b.  Source  of  the  Operational  Suitability  Criteria. 

What  is  the  source  of  the  quantitative  and  qualitative  Ciiteria  for  the  operational  suitability 
elements? 

When  the  criteria  in  the  operational  suitability  are  stated  in  the  test  report,  they  give  die  reader  a 
fiamewotic  fm  evaluation  ot  the  test  results.  To  provide  a  complete  picture,  the  test  report  should 
identify  what  the  sources  were  for  the  evaluation  criteria.  This  is  particularly  important  in  the 
statement  of  any  qualitative  criteria.  Most  quantitative  criteria  can  be  traced  to  requirements 
documents,  but  this  is  not  always  true  of  the  qualitative  criteria.  (OFTEVFOR  does  not  use 
criteria  on  the  qualitative  measures.) 


The  system  reliability,  maintainability,  and  availability  requirements  were 
specified  in  the  July  1987  revision  of  the  system  Required  Operational 
Capability  (ROC). 
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c.  Quantitative  Reliability,  AvailaMiity,  and  Mamtainability  (RAM)  Measures. 

Are  the  smtabiUty  characteristics  for  RAM  stated  in  quantitative  terms? 

For  most  systems,  qjciadonal  suitability  characteiistics  such  as  reliability,  avail^iility,  and 
maintainahility  (RAM)  can  be  expressed  with  quantitative  parameters.  The  evaluation  criteria 
for  these  areas  therefore  should  be  stated  quantitatively  and  used  in  evaluating  the  test  results. 
The  definititHi  of  the  appropriate  terminology  is  an  essential  part  of  understanding  diese 
numerical  values. 


The  fixed-installation  communication  set  was  required  to  have  a  mean  time 
between  operationat  mission  failure  (MTBOMF)  of  greater  than  500  hours. 


d.  ConCdence  Limits. 

Are  tliere  statements  of  confidence  limits  for  the  RAM  quantitative  criteria? 

The  measurement  of  statistical  parameters  such  as  reliability  can  require  considerable  test  time. 
If  limited  test  time  is  available,  as  is  often  die  case,  then  the  measured  value  is  expiessed  with  a 
degree  of  statistical  confidmee.  Background  information  on  detennining  confidence  levels  firom 
test  data  is  discussed  in  DoD  323S.1-H,  Tea  and  Evaluation  System  Reliability,  Availability, 
and  Maintainahiliiy-A  Primer.''  This  document  discusses  tbe  mi^iematics  of  test  statistics,  but 
does  not  aid  in  deriding  what  level  of  confidence  is  needed.  (While  (VTEVFC^  uses 
confidence  calculations  in  test  design,  these  factors  usually  are  not  mentioned  in  their  test 
rqiotts.  Tbe  other  OTAs  use  confided  levels  when  they  arc  consideicd  appropriate.) 


The  OTiE  report  must  summarize  the  actual  suitability  testing  that  was 
performed.  This  summary  must  Include  an  adequate  description  of  the 
operational  suitability  testing  and  present  all  significant  changes  from 
Information  that  was  presented  In  the  OT&E  plan. 

AREA  OF  RISK 

Omittiiig  critical  infomiatioii  may  alter  the  meaning  of  the  test  results. 

A  change  in  die  planned  suitability  test  activity  may  result  in  significant  changes  to  the  meaning 
of  the  test  results.  The  test  report  should  present  the  description  of  what  was  actually  done  and 
what  the  results  were.  Some  of  the  information  on  deficiencies  may  be  omitted  fiom  the 
summary  in  an  effort  to  condense  the  information  or  to  pn^t  the  results  from  further  scrutiny. 
The  DOT&E  Staff  Assistant  must  have  enough  visibility  into  the  actual  testing  to  know  if  tte 
summary  is  complete  and  accurately  reflects  the  suitabili^  T&E  that  was  performed.  The  risk  in 
the  summary  is  that  some  critical  itdormation  will  not  be  included. 

OUTLINE  FOR  REDUCING  RISK 

a.  Comparisan  to  Test  Plan. 

Was  the  operational  suitability  test  conducted  consistent  with  the  suitability  test  planning? 

The  discussion  should  highlight  any  differences  between  the  suitability  test  as  planned  and  as 
conducted.  Any  differences  most  likely  are  at  the  detailed  level,  thus,  some  det^ed  discussion 
is  required  to  point  out  the  differences  and  any  uiqiact  these  difierences  may  have  on  the  test 
results. 

When  cqierational  testing  is  perfcamed,  there  often  ate  changes  required  because  of  conditions 
that  were  not  foreseen  when  the  OT&E  plan  was  prepared.  These  changes  must  be  summarized 
in  the  OT&£  rqKA  They  should  be  discussed  in  enough  detail  to  permit  die  reader  to  assess 
what  inqiact  thex  items  h^  on  the  test  and  the  test  results. 


b.  Unexpected  Limitations. 

Were  there  any  unexpected  limitations  to  the  operational  suitability  portion  of  the  test? 


UDe]q)ected  test  limitatioiis  may  result  £nom  the  diffoeoces  discussed  above,  but  they  also  may 
result  &om  unexpected  suitability-related  Actors,  such  as  weather  and  support  personneL 


The  results  of  the  operational  testing  showed  that  the  maintenance  training  that 
was  conducted  prior  to  the  start  of  the  test  was  inadequate  in  the  area  of  radar 
fault  isolation  and  troubleshooting.  This  deficiency  preciuded  a  complete 
evaluation  of  maintainability,  maintenance  documentation,  and  the  diagnostics 
system  for  the  radar. 


c.  Summary  of  the  Test  Performed. 

Is  the  summary  of  the  test  that  was  performed  complete  and  unambiguous? 

Based  upon  ’Ttrst-lumd"  knowledge  of  the  test  conduct,  die  Staff  Assistant  should  assess  die 
summary  in  the  OT&E  report  Different  authors  wW  describe  smtabdity-relaled  events 
diffaendy.  Mendcal  summary  descriptions  are  unlikely,  but  the  objective  is  to  lave  a  discussion 
diat  includes  a  fair  and  accurate  summary  of  the  sigmficant  aspects  of  die  st^abOity  testing  diat 
was  conducted.  The  facts  cooceming  what  was  done  and  w^  occurred  daring  the  suitf^ty 
testing  should  be  clearly  stated.  The  activities  should  be  related  to  die  test  planning  or  odicT 
reference  documents. 


The  summaiy  of  the  test  results  Is  the  msior  section  of  the  OT&E  test  report  The 
Information  here  should  support  the  suitability  condus/ons  reached  and  provide 
a  basis  for  the  readers  to  form  Judgments  that  agree  vifth  the  iru^r  conclusions. 
The  level  of  detail  that  Is  pmvided  should  permit  the  readers  to  Integrate 
Independent  thoughts  with  the  details  of  dtesultMIhy  test  results. 

AREA  OF  RISK 

Poor  presaitatioa  of  important  suitability  factors  may  lead  to  incorrect  condnsions. 

The  test  lesults  must  be  discussed  in  enough  detail  to  support  any  conclusions  that  die  system 
meets  or  does  not  meet  its  suitability  criteria  for  die  phase  being  assessed.  If  adequate  dt^  is 
not  presented,  die  majw  conclusions  may  not  be  accepted  and  may  not  be  supported  by  the 
teadtf .  The  major  findings  should  be  hi^ghted  in  a  manner  that  gives  the  read^  insight  into 
the  important  suitability  results  that  evtd^  fnm  the  test  and  the  evaluadon  of  the  test  da^ 

OUTLINE  FOR  REDUCING  RISK 

a.  Presentation  of  Suitability  Results. 

Are  the  major  suitabiiity  findings  presented  in  an  understandable  nay? 

The  major  operational  suitability  finding^  should  be  in  areas  related  to  the  suitalnlity  critical 
tqieiatir^  issues  (CX>Is).  Tbetc  may  be  in^portant  operational  suitability  findings  in  odier  areas 
as  weU.  The  DOT&E  Staff  Assistant  needs  to  review  the  findings  and  ensure  tto  all  inqxxtant 
areas  are  addressed. 


Overall  Evaluation:  The  operational  suitability  oi  the  system  is  satisfactory. 
Reliability  and  maintainability  exhibited  in  OT&E  exceeded  the  stated 
requirements. 
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b.  Projection  Methods. 

Was  the  method  used  to  project  RAM  values  from  the  test  results  to  mature  values 
examined  and  validated? 

If  the  test  report  states  values  to  the  matuie  projection  of  reliability  and  maintainability  (R&M), 
is  the  method  of  projection  described  in  enough  detail  to  evaluate  its  ^plicability?  Some 
operational  testing  agencies  never  use  projectimis,  others  use  them  often.  There  are  several 
ptojecticMi  methods  available  for  use.  When  they  are  used,  tiie  validity  of  die  methods  and 
applicability  of  die  meAod  to  die  situation  at  hand  must  be  assessed.  The  use  of  a  questionable 
projection  method  may  cause  the  decisionmaker  to  place  little  value  on  the  OT&E  results.  A 
projection  should  never  be  reported  as  a  test  result— die  test  result  should  be  an  observed  value. 


The  reliability  test  results  were  projected  to  the  mature  system  values  using 
methods  from  Military  Handbook  183.  The  applicability  o<  the  method  to  the 
system  was  validated  by  using  growth  experience  with  similar  systems,  and  by 
reviewing  the  reliability  growth  plans  presented  by  the  program  manager. 


c.  Confidence  Levels. 

Are  confidence  levels  stated  for  the  quantitative  R&M  values  that  resulted  from  the  tests? 

The  use  of  limited  test  data  to  provide  measures  of  reliability  and  maintainability  always  includes 
room  for  statistical  etrtM' in  the  quantitative  estimate.  The  likelihood  of  the  eiror  is  shown  by  the 
statistical  confidence  associated  with  the  test  tneasmtment  For  «Kdi  appopiiate  measurement, 
a  statistical  confidence  should  be  stated.  (While  CffTEVFOR  uses  co^dence  calculations  in 
both  test  design  and  in  evaluation,  the  confidence  values  arc  nm  included  in  OFTEVFOR  test 
reports.) 


Reliability:  An  80  percent  lower  one-sided  confidence  interval  was  calculated 
tor  the  point  estimate  value  of  the  MT80MF.  Operational  mission  failures  were 
dete-mned  by  a  formal  scoring  conference  in  accordance  with  the  approved 
failuio  definition  and  scoring  criteria.  The  80  percent  confidence  value  exceeded 
the  requirement  stated  in  the  approved  ROC. 


d.  Impact  of  Suitability  Results. 

Is  the  impact  or  consequoice  of  the  major  suitability  findings  stated? 

When  operatitmal  suitability  results  are  cooqiared  to  their  crite^  the  tqiott  should  state  diat  for 
'  this  particular  suitability  area,  the  system  is  accqitable,  marginal,  unatxxptable,  or  make  some 
other  judgment  The  impact  or  consequence  diese  conditions  on  system  should  be 
ioclud^  in  the  test  rqiott  The  DOT&E  Staff  Assistant  should  have  an  onderstanding  of  tiie 
>  situation  and  its  impact  on  tiie  system's  opeiati<»al  capalnlity. 


The  stability  of  the  ammunition  trailer  is  marginal.  The  implication  of  this 
deficiency  is  that  the  trailer  will  be  unable  to  cany  a  full  load  of  ammunition  and 
travel  at  normal  speeds  on  an  unpaved  road  without  tipping  over. 
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The  conclusions  that  an  nached  as  a  nsult  ol  the  operational  testing  shouU  be 
clearly  stated  In  the  OT&E  report  In  this  section,  then  should  be  a  dear 
statemerrt  of  whether  tite  system  was  conskfeted  operationally  suitable  or  not. 

AREA  OF  RISK 

Pooriy  slated  or  onuttedcondusioiis  can  result  in  cntmcous  management  decisions. 

Not  an  test  reports  contain  concluaons  on  the  acceptability  of  the  system.  Conclusions  should 
be  stated  that  indirate  the  major  points  that  are  drawn  fiom  the  suitability  test  results  and  die 
evaluation  of  die  data  against  test  criteria.  Was  the  test  passed  and  what  did  the  test  prove? 
What  areas  are  acceptable  and  which  are  not?  What  is  the  meaning  of  the  deficient  areas  relative 
to  die  progress  of  the  system  toward  its  suitability  for  operational  use? 

OUTLINE  FOR  REDUCING  RISK 

a.  Operational  SmtabOity  Condnsion. 

Docs  the  report  state  whether  the  system  is  considered  operationally  snilahle  or  not?  Is 
this  operational  snitahility  assessment  a  composite  view  of  the  soitalnlity  dements? 

How  does  the  rqxxt  deal  wadi  the  situation  where  some  of  the  elements  are  satisfactory  and 
some  are  not  satisDctoiy?  Are  the  unsatisfactory  items  highlighted?  Those  items  that  are  not 
widiin  the  uso's  stated  needs  or  diat  do  not  meet  the  OT&E  criteria  must  be  assessed  for  impact 
on  the  system's  overall  suitability  for  fielding.  A  single  conclusion  in  the  operational  suitahffity 
area  may  be  difficult.  This  teriures  die  comlnnation  of  die  many  suitalnli^  demmts,  some  of 
which  may  be  good,  some  margiiiai,  aid  some  not  evaluated,  into  a  angle  judgrrmt  The 
suitability  test  r^ts  must  be  evduat^  within  die  context  of  the  planned  operatioiial  use  of  the 
system  rad  a  judgment  made  as  to  whether  the  system  meets  the  user's  nee^.  Suitability  COIs 
are  used  to  focus  attention  on  areas  diat  should  receive  increased  weighting  in  any  total 
assessment 
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b.  Operalioiial  Effects  of  Any  Adverse  Test  Results. 

What  is  the  operational  implication  of  the  operational  suitahility  deficiencies? 

When  an  qietational  suitability  deficiency  is  highlighted  in  the  test  report  conclusions,  the  rqwn 
should  include  die  in^licadon  of  diis  deficiency  on  the  operadtmal  use  of  the  systeia  How  is 
die  system  limited  in  its  use?  What  missions  or  uses  are  degraded  by  the  deficiency?  What 
additional  resouices  ate  requited  to  compensate  for  the  existence  of  the  ^ficiency? 
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